Hadoop HDFS Reviews & Product Details

HDFS is Hadoop distributed File system. The best thing I like about HDFS is reliability I get with Hadoop, its file replication is great and there are very less chance of your data being lost. To get the best benefits out of hadoop keep the file size big. At least 100MB each file. Then you will realize the power of Hadoop. Fault tolerant file system etc. Review collected by and hosted on G2.com.

Its a little bit slower, but then what is not slow when you come to big file systems which work with Tera bytes of data. Even Amazon S3 is extremely slow when reading the data from it. Other than that it might be little tricky to find documentation for new users / features. Review collected by and hosted on G2.com.

Well...HDFS files are write once files. I consider HDFS files as write-once and read-many files. There is no concept of random writes. It is optimized for streaming access of large files. I would typically store files that are in the 100s of MB upwards on HDFS and access them through MapReduce to process them in batch mode. Review collected by and hosted on G2.com.

HDFS doesn't do random reads very well. A caveat of HDFS to remember, it is a distributed file system abstracted on top of local file system by hadoop, suitable for storing huge files; however, it does not provide facility of tabular form of storage as such. Review collected by and hosted on G2.com.

Setting up a 2 node Hadoop cluster was easy. We just used it as a distributed storage medium. Review collected by and hosted on G2.com.

What is the hype behind this? One could easily set up a 2 node HDFS and say a product was conformant. Review collected by and hosted on G2.com.

All type of tool on top of HDFS like pig/hive (help to reduce your time in writing MR jobs), sqoop (to transfer data from RDBMS <-> hdfs), NOSQL db can use as their storage FS (ex. HBASE) and many more are available currently plus many big organizations (cloudera, MAPR, hortonwork to name a few) are actively contributing in the HDFS ecosystem. Review collected by and hosted on G2.com.

For beginners/first-timers, it is a bit difficult to set up HDFS in cluster mode, and currently, HDFS uses YARN2 as their resource manager, which is developed with very narrow thinking. They can enhance it by looking at MESOS. Every stage data (intermediate result) is stored on disk, for streaming processing HDFS is the worst choice. Review collected by and hosted on G2.com.

Automatic replication, stable, compatible with/required for the rest of the Hadoop ecosystem. Its pretty easy to manage. Rack Awarenes ensures that losing a single rack doesn't result in the loss of data. Overall, its kind of an awkward thing to write a review about since, it is really an enabling-technology. However, when combined with an analytical tool that can take advantage of HDFS (like Map/Reduce, Hive, Pig, etc), HDFS shows its value. I'm also not aware of any alternatives to using HDFS with these tools. Review collected by and hosted on G2.com.

Its purpose-built for Hadoop, and large-scale data processing. That said, it doesn't really work well as a general-purpose filesystem. You can mount it with NFS, but realize that as bad as NFS is, NFS on HDFS is worse. Don't get caught in this trap. For the most part, stick to interacting with it using Hadoop's tools, not generic filesystem tools over NFS. Review collected by and hosted on G2.com.

After working with Hadoop for 6 years, I like the direction in which it has evolved. It started as a tightly-coupled offering of a distributed data platform (HDFS) and an analytics processing framework (MapReduce); however, it has since expanding its scope tremendously. After the initial success of MapReduce, it has become quite clear that it has many limitations as an algorithm. Other frameworks such as Apache Spark have far surpassed its capabilities. Seeing that trend, the Hadoop team instead chose to focus on Hadoop as a base platform for dozens of data and analytics offerings. This lead to the strengthening of the already robust HDFS and the creation of YARN as an applications framework. This gives the Hadoop platform a lot of widespread appeal and makes it a great basis for any big data processing platform. Many tools exist for quickly standing up entire Hadoop clusters in very little time, and it treats scaling and fault tolerance as primary as first-order priorities. Review collected by and hosted on G2.com.

There are two major areas that Hadoop could use improvement that have existed since the beginning and continue to be a problem for the implementation of Hadoop in real world settings. The first is poor documentation on performance and tuning. Hadoop works fairly well out of the box, but once you start to encounter problems there are very scarce resources on trying to troubleshoot those issues. Hadoop has been around long enough as an open source project that common configuration strategies and troubleshooting techniques should be built into the documentation. Secondly, and more importantly for many users, are the lack of security options for Hadoop. There are few if any built-in options, and any plugable solutions are fairly difficult to implement. Review collected by and hosted on G2.com.

I like MapReduce code a lot. Mappers and Reducers and how the overall hierarchy goes by. Hadoop is a platform that I have chosen a year ago and still I am in love with it because of its simplicity for solving complex problems involving very large databases. I also did Apache Giraph which goes into graph processing and was a great experience learning a whole new product of Hadoop. I like solving real-life challenges using Hadoop like predicting earthquakes so that the results could be less devastating. This is one of my proposed ideas but there are many more like these which I like a lot. Review collected by and hosted on G2.com.

I dislike the numerous products that are developing in Hadoop architecture because a developer can never learn all the products based on Hadoop. He/She can learn only a few ones which are used in a very extensive way. So why not integrate other less used products in the mostly used products so that it gets the added functionality. Review collected by and hosted on G2.com.

It's currently the best distributed file system for implementing biodata projects. The main reason rely on the fact that HDFS is fully integrated with many parallel computing platforms for doing BigData analysis: Map/Reduce, Spark, Impala, Drill. It's very mature, stable and really robust, optimised for streaming data to the application layer. It provides now a full support for security, Posix-like, Posix ACLS and support for directory based encryption. Review collected by and hosted on G2.com.

It works very badly with many small files. HDFS is optimised for dealing with a relatively small number of files but very big. This is an annoying limitation that forces many architectural decisions for supporting the data ingestion effectively. Review collected by and hosted on G2.com.

It is quick and perform well with small clusters too. I used it to implement my thesis. Could be a little challenging if u not aware with linux Review collected by and hosted on G2.com.

The debugging and logging is not very user friendly. Although it has a decent interface for job tracking. Review collected by and hosted on G2.com.

Hadoop HDFS is proven scalable and stable enough for big data processing. I have over 6 years in product development and operation on HDFS. Storing and processing terabytes scale data with HDFS. Hadoop HDFS handles scale problem well and most of problem can be solved. Review collected by and hosted on G2.com.

HDFS is optimized for big file and batch oriented data process. User should really need to pay attention on "Small File Problem", avoiding produce large amount of small files. This will eventually kill HDFS. Review collected by and hosted on G2.com.