5 Most Active Apache Big Data Projects

You may have heard of this Apache Hadoop thing, used for Big Data processing along with associated projects like Apache Spark, the new shiny toy in the open source movement. They're among the most active and popular projects under the direction of the Apache Software Foundation (ASF), a non-profit open source steward.

But you may not have heard of lesser-known Big Data projects guided by the ASF that can be equally useful in your development efforts -- projects like Bigtop, REEF and Storm.

Here's a list of the five most active projects listed by the ASF under the "Big Data" category, ranked by a combination of the number of committers and the number of associated Project Management Committee (PMC) members.

Apache Spark
The No. 1 project is the aforementioned Apache Spark. (Strangely, Hadoop is classified under the "Database" category, not "Big Data.") It's probably very familiar, having been covered extensively on this site and virtually every other tech-oriented media outlet. With more than 1,000 code contributors last year, it's been ranked among the most active of ASF projects and open source projects in general.

Described as "fast and general engine for large-scale data processing," it's a red-hot rising star in the open source Big Data ecosystem, bringing performance and superior streaming analytics functionality that surpasses original Hadoop ecosystem components such as MapReduce. It powers libraries such as Spark SQL, Spark Streaming, MLib (machine learning) and GraphX (an API for graphs and graph-parallel computation). With full-on adoption by companies such as IBM, MapR Technologies and Databricks, its committers come from 16 different organizations.

Last year, IBM hopped on the Spark bandwagon in a big way -- promising to put more than 3,500 researchers and developers to work on related projects at labs around the world -- while calling it "potentially the most significant open source project of the next decade."

The Open HUB site lists 31,376 commits made by 1,139 contributors who wrote 821,016 lines of code. The main contributor is Reynold S. Xin, a Databricks developer. The project "took an estimated 228 years of effort (COCOMO model) starting with its first commit in March, 2010 ending with its most recent commit 14 days ago," says Open HUB, an online community and public directory of free and open source software operated by Black Duck Software.

The Spark GitHub site lists 16,001 commits coming from 875 contributors, with Matei Zaharia coming in a close second behind fellow Databricks colleague Xin.

Spark is programmed in Java, Scala and Python, with 44 committers and 35 PMC members. Having moved to version 1.6.1 in March, it can be downloaded here.

Apache Tez
Next up is Tez. (Actually, CouchDB would be No. 2, but I see it belonging to the "Database" category. I was tipped off by this official description: "Apache CouchDB is a database ....")

Tez "is aimed at building an application framework which allows for a complex directed-acyclic-graph of tasks for processing data. It is currently built atop Apache Hadoop YARN." Tez also improves upon the performance of the much-maligned MapReduce, while featuring resource management, simplified deployment and dynamic physical data flow decisions, among many other attributes.

"By allowing projects like Apache Hive and Apache Pig to run a complex DAG [directed acyclic graph] of tasks, Tez can be used to process data, that earlier took multiple MR [MapReduce] jobs, now in a single Tez job," the project site states.

"Apache Tez provides a developer API and framework to write native YARN applications that bridge the spectrum of interactive and batch workloads," Hortonworks says. "It allows those data access applications to work with petabytes of data over thousands nodes. The Apache Tez component library allows developers to create Hadoop applications that integrate natively with Apache Hadoop YARN and perform well within mixed workload clusters."

According to the answer to a Quora question about Tez vs. Spark, a Cloudera developer replied: "Both Tez and Spark provide a distributed execution engine that can handle arbitrary DAGs, targeted towards processing large amounts of data. Both can read and write data to and from Hadoop using any MapReduce input or output format. The main focus of Tez so far has been providing a faster engine than MapReduce under Hadoop's traditional data-processing languages like Hive and Pig."

The Open HUB site says Tez features 185,000 lines of code, coming from 12 current contributors, with the top contributor during the last year listed as Siddharth Seth.

The Tez GitHub site lists 2,136 commits from 15 contributors, led by Hitesh Shah, a Hortonworks developer.

Tez is programmed in Java, sports 34 committers and has 32 members on its PMC. Currently in version 0.8.3, it can be downloaded here.

Apache Bigtop
"Bigtop is ... for infrastructure engineers and data scientists looking for comprehensive packaging, testing and configuration of the leading open source Big Data components," the project site states. "Bigtop supports a wide range of components/projects, including, but not limited to, Hadoop, HBase and Spark."

According to Bigtop's GitHub site, "The primary goal of Apache Bigtop is to build a community around the packaging and interoperability testing of Apache Hadoop-related projects. This includes testing at various levels (packaging, platform, runtime, upgrade, etc...) developed by a community with a focus on the system as a whole, rather than individual projects."

Running on Linux variants Debian, Ubuntu, CentOS, Fedora, openSUSE and many others, Bigtop has been called "the Fedora of Hadoop" by Cloudera.

"I have been using Apache Bigtop to build selected Hadoop projects customized for my particular needs, developer Christian Tzolov wrote in his technical blog last summer. For example to build RPMs for the newest Spark release or to build specific version of Hue, Flume, Sqoop ... suitable for my Hadoop distro. One can use it to explore incubator projects like Apache Zeppelin and so on."

With the project's affinity for virtualization ("Bigtop provides vagrant recipes, raw images and (work-in-progress) Docker recipes for deploying Hadoop from zero."), Tzolov details using it with a Docker CentOS image.

The Open HUB site says Bigtop is made up of 146,000 lines of code, with 41 current contributors, led by a mysterious "cOs."

The Bigtop GitHub site lists 1,709 commits stemming from 66 contributors, with Pivotal developer Roman V. Shaposhnik (379 commits) actually leading "cOs" (226 commits).

Bigtop is programmed in Java, with 32 committers and 24 PMC members. Having hit version 1.1.0 last January, it can be downloaded here.

Apache REEF
"REEF, the Retainable Evaluator Execution Framework, is our approach to simplify and unify the lower layers of Big Data systems on modern resource managers," the project's site states.

With the goal of eventually maturing into a Big Data Application Server, "REEF provides a centralized control plane abstraction that can be used to build a decentralized data plane for supporting Big Data systems. Special consideration is given to graph computation and machine learning applications, both of which require data retention on allocated resources to execute multiple passes over the data."

REEF stemmed from the increased programming framework capabilities realized when the YARN resource management software evolved to version 2, being decoupled from the limited MapReduce programming model.

"It is well understood that while enticingly simple and fault-tolerant, the MapReduce model is not ideal for many applications, especially iterative or recursive workloads like machine learning and graph processing, and those that tremendously benefit from main memory (as opposed to disk based) computation," the REEF FAQ states. "A variety of Big Data systems stem from this insight: Microsoft’s Dryad, Apache Spark, Google’s Pregel, CMU’s GraphLab and UCI’s AsterixDB, to name a few. Each of these systems add unique capabilities, but form islands around key functionalities, making it hard to share both data and compute resources between them. YARN, and related resource managers, move us one step closer toward a unified Big Data system stack. The goal of REEF is to provide the next level of detail in this layering."

REEF originated at Microsoft as an effort to improve Big Data application development with any combination of JVM and .NET languages. Having been open sourced in 2013, it's still in heavy use at the Redmond software giant. "Today, REEF forms the backbone of production services such as Azure Stream Analytics and attracts contributions from other teams both from Microsoft and outside," says a Microsoft TechNet blog post from last December, when it graduated to an ASF top-level project. "Further, REEF enables research at several top-tier universities, including at University of Washington, TU Berlin and Seoul National University."

The Open HUB site says REEF has had 3,434 code changes committed by 52 contributors, led by Andrew Chung, who together provided 157,881 lines of code. "Over the past 12 months, 33 developers contributed new code to Apache REEF," the site states. "This is one of the largest open source teams in the world, and is in the top 2 percent of all project teams on Open Hub."

The REEF GitHub site lists the same number of commits, but only 34 contributors, led by Microsoft's Markus Weimer.

REEF is programmed in Java, C# and C++, with 32 committers listed by the ASF and 21 PMC members. Now at version 0.14.0, it can be downloaded here.

Apache Storm
Crashing upon the windswept REEF, Storm is ... sorry, couldn't help it. Storm is actually "a free and open source distributed real-time computation system," according to its Web site. "Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!"

Boasting fast processing speeds, Storm can be applied to many varied tasks, including real-time stream-processing analytics jobs such as data normalization; server event log monitoring/auditing; online machine learning; continuous computation; and distributed Remote Procedure Calls {RPC), Extract, Transform and Load (ETL); and more.

In answer to a Quora question, FPT Telecom engineer Trie Tan Nguyen said he uses it with Redis, Kafka and Netty to perform real-time analytics. Groupon reportedly uses it to develop real-time data integration systems, and it also is said to power Twitter's publisher analytics, logging every tweet and click to provide raw data to Twitter's publishing partners.

Storm was used to power the original StreamAnalytix software at Impetus Technologies, which in February announced it was adding Spark support to its product. That same month, OpsClarify announced it was supporting Storm -- along with Spark and Kafka -- in its Intelligent Monitoring solution.

The project's Open HUB site says Storm comprises 301,000 lines of code, supplied by 125 current contributors. Robert (Bobby) Evans is listed as the top all-time contributor with 974 commits.

The associated GitHub site lists 6,801 commits from 208 contributors, led by Nathan Marz, a former Twitter developer now reportedly running a stealthy start-up.

Storm is programmed in Java, with 29 committers listed by the ASF and 28 PMC members. Currently at version 0.9.5, it can be downloaded here.

And More
There are many more popular ASF projects related to Big Data, of course, such as Hive, Pig, HBase and others, but, again, the ASF lists them under the "Database" category.

Here I've just examined the most active "Big Data" projects. Others that might deserve your developer consideration, rounding out the top 10, include: Airavata, Drill, Ignite, Phoenix and Scoop, among many others.

Weigh in with your thoughts about open source Big Data projects by commenting here or dropping me a line.

Posted by David Ramel on May 3, 20160 comments

Share

• LinkedIn
• Facebook
• Twitter

JavaScript Style: Semicolons, or No?

A fledgling effort to promote a new standard for styling JavaScript code is getting hung up on the issue of whether or not to use semicolons over at Hacker News.

The JavaScript Standard Style project on GitHub (and here on the Web) promises "one style to rule them all."

"The beauty of JavaScript Standard Style is that it's simple," the project's README.md states. "No one wants to maintain multiple hundred-line style configuration files for every module/project they work on. Enough of this madness! This module saves you time in two ways:

• No configuration. The easiest way to enforce consistent style in your project. Just drop it in.
• Catch style errors before they're submitted in PRs. Saves precious code review time by eliminating back-and-forth between maintainer and contributor."

Although it appears to have been in the works for about a year, it was just introduced on the coding forum site Hacker News today, where semicolons emerged as a big sticking point, judging from comments such as:

• Why do these people want to avoid semicolons at all costs? Even inconsistencies? Even potential bugs? Why?? I just don't get it.
• If there was an actual gain to leaving out the semicolons, I'd be onboard but the only gain here is someone's aesthetic pleasure and someone else's aesthetic displeasure.
• No semicolons, two space indentation, space after function name, single quotes for strings? There's no way this would fly with a majority of developers.
• The no semicolons and space after function name are really the weird things here.
• I started skimming the rules and was like "this all seems fine" and then noticed "no semicolons". YOU'LL PRY THEM FROM MY COLD, DEAD HANDS FIRST.
• no semicolons = no deal.
• There's no real good reason why, but my mild OCDish tendencies get intensely triggered by looking at Javascript with no semicolons.
• Downside [of adopting this style, which the commenter did] is that very few other authors are using it, so when you share code people are concerned about its format (no semicolons in particular).
• No semicolons? REALLY? We've already spoke about this, people.

Ah, yes, spoken about it we have. It turns out this debate has been raging in the JavaScript world for years -- possibly since JavaScript creator Brendan Eich first called it a wrap in 1995.

"Do you recommend using semicolons after every statement in JavaScript?" asks a question on the coding Q&A site StackOverflow seven years ago.

Even earlier than that, Robert Nyman addressed the issue in a post that generated debate in comments that persisted for some four years. (His advice: "make sure to always put semicolons in your code, to avoid potential problems."

Eich himself addressed the issue in a 2012 post titled The Infernal Semicolon. "Most of the comments in this semicolons in JS exchange make me sad," it starts out.

And Hacker News itself published a poll on the question exactly 2,103 days ago. Answers to "Poll: Semicolons in JavaScript" were 418 points yea and 45 points nay.

From a quick perusal of the debate still underway on the Web, that seems to be in line with the general developer consensus on the matter.

Not that there aren't vocal supporters of doing away with semicolons, even in the aforementioned Hacker News post. "Everything will be fine if you don't use semicolons," read one comment. "My team and I haven't, for years now. It's really, really, truly honestly ok. I promise. (Only exception is for-loops)."

The creators (Feross Aboukhadijeh published the GitHub effort) of this new style effort echo that view and obviously anticipated the ensuing debate. The project rules specify: "No semicolons -– It's fine. Really!" (Note there are three different links in the preceding text, though it might look like just one.)

The "official" stance is somewhat murkier. Ecma International is in charge of setting standards for the scripting-language specification called ECMAScript, under which fall scripting languages such as JavaScript, JScript and ActionScript.

Here's what that governing body says in a section titled "Automatic Semicolon Insertion":

Certain ECMAScript statements (empty statement, variable statement, expression statement, do-while statement, continue statement, break statement, return statement, and throw statement) must be terminated with semicolons. Such semicolons may always appear explicitly in the source text. For convenience, however, such semicolons may be omitted from the source text in certain situations. These situations are described by saying that semicolons are automatically inserted into the source code token stream in those situations."

There's a bunch more verbiage on the rules for Automatic Semicolon Insertion, but I didn't even try to decipher it as I don't speak ECMAScript-ese. If somebody wants to wade through that officious-sounding crap and provide me with a TL:DR, I'm all ears. (One sentence in this Automatic Semicolon Insertion "guidance" is 88 words long -- who writes this stuff?)

A GitHub project detailing ECMAScript 6 features provides more information in something more closely resembling English:

ECMAScript since its earliest days supported automatic semicolon inference/insertion, of course. But people coding ECMAScript 5 started it in an era where lots of tools (especially source code compressors) had problems when semicolons were left out from the source code. As a consequence, most ECMAScript 5 coders maintained the traditional coding style with semicolons as the syntactic sugar, although in most of the cases they are not necessary. But this era is gone today. Both ECMAScript 6 and all tools (including compressors) perfectly support automatic semicolon inference/insertion nowadays. As a consequence, ECMAScript 6 coders nowadays can get rid of nearly all semicolons and remove clutter from their source code.

I'm all for removing clutter, but I somehow can't see myself not using semicolons. They just seem to provide some structured stability or something in a language that's fast and loose enough as it is.

What about you? Weigh in here in the comments or drop me a line.

Posted by David Ramel on April 28, 20160 comments

Share

• LinkedIn
• Facebook
• Twitter

Microsoft, Facebook Lead the Way in New-Age Bot Programming

As chatbot Tay could tell you (hopefully without racist bigotry), bots are a hot topic in software development right now, with Microsoft and Facebook among the most prominent leaders guiding the movement.

While they have recently seen a marked uptick in interest assisted by artificial intelligence (AI), machine learning (ML) and cognitive computing advances, bots have been around for a while -- and have been stirring things up in unexpected ways at both Microsoft and Facebook.

Though some early efforts in the evolving arena of AI-assisted bots are obviously -- and understandingly -- rough around the edges, the two companies are providing tools and guidance for developers wanting to explore new bot-enabled opportunities.

The importance of the evolving technology to both companies was shown recently as both Microsoft CEO Satya Nadella and Facebook CEO Mark Zuckerberg featured bots -- specifically chatbots -- extensively in keynote addresses at their respective developer conferences.

Nadella framed the bots conversation in the context of a "Conversations as Platform" initiative.

"We also are going to talk about a new emerging platform," Nadella said. "We're in the very early days of this. It's actually at the intersection of all of our three ambitions. We call it Conversation as a Platform. It's a simple concept, yet it's very powerful in its impact. It is about taking the power of human language and applying it more pervasively to all of our computing. That means we need to infuse into our computers and computing intelligence, intelligence about us and our context. And by doing so we think this can have as profound an impact as the previous platform shifts have had, whether it be GUI, whether it be the Web or touch on mobile. And so we'll have an extensive conversation about the beginning of this new platform journey."

Just a couple weeks later, Zuckerberg echoed that theme at the company's F8 developer conference.

"We think that you should just be able to message a business in the same way you message a friend," Zuckerberg said. "You should get a quick response and it shouldn't take your full attention like a phone call would and you shouldn't have to install a new app. So today we are launching Messenger Platform, so you can build bots for Messenger. And it's a simple platform that's powered by artificial intelligence, so you can build natural language services to communicate directly with people."

Bots at Microsoft
As for the tools and guidance, Microsoft introduced the Microsoft Bot Framework preview to "build and connect intelligent bots to interact with your users naturally wherever they are, from text/SMS to Skype, Slack, Office 365 mail and other popular services."

It features a Bot Connector to help developers connect their existing bots to different communication tools, such as GroupMe, SMS, e-mail and so on. For creating brand-new bots, there's a Bot Builder for Node.js, with a BotBuilder SDK hosted on GitHub.

The company is also showcasing sample bots that were shown off at Build in a Bot Directory. These include bots that summarize articles from provided URLs, search Bing for news, images and music, explore "what if" scenarios and more.

Just a few days ago, Microsoft expanded client support for its Skype bot, part of the company's Skype Developer Platform that invites developers to "Build a bot for Skype. Get access to a preview of the Skype Bot Platform."

Earlier in the year, the company's research unit introduced Botivist, "a platform that uses Twitter bots to find potential volunteers and request contributions."

Earlier this month, Microsoft's Jeff Hollan published a blog post detailing how the Bot Framework works better with the company's Azure cloud service. "In just one weekend I was able to make a bot that leveraged workflows (Logic Apps), APIs (Web Apps), security (Azure AD), on-premise systems (Hybrid Connections), and Machine Learning (luis.ai) to create a bot that improved my work productivity. The power and variety of Azure offerings was the only reason I was able to build such a system in such a short amount of time."

Bots also figure prominently in Microsoft's Cortana Intelligence Suite, which includes Microsoft Cognitive Services, "a collection of intelligence APIs that allows systems to see, hear, speak, understand and interpret our needs using natural methods of communication."

"Bots (or conversation agents) are rapidly becoming an integral part of one's digital experience -- they are as vital a way for users to interact with a service or application as is a Web site or a mobile experience," Microsoft says in its Bot Framework documentation. "Developers writing bots all face the same problems: bots require basic I/O; they must have language and dialog skills; and they must connect to users -- preferably in any conversation experience and language the user chooses. The Bot Framework provides tools to easily solve these problems and more for developers [for example], automatic translation to more than 30 languages, user and conversation state management, debugging tools, an embeddable web chat control and a way for users to discover, try and add bots to the conversation experiences they love."

Bots at Facebook
Over at Facebook, the bot initiatives aren't as extensive, with a more targeted focus on its beta Messenger Platform, which promises to help developers "discover all the ways bots on the Messenger Platform help you join the conversation." Earlier this month, the company published a guide on How To Build Bots for Messenger.

"Starting today, all developers and businesses will be able to build bots for Messenger, and then submit them for review," Facebook's Seth Rosenberg said. "We will gradually accept and approve submissions to ensure the best experiences for everyone on Messenger. We're putting people first with new guidelines, policies and controls to offer the best interactions we possibly can."

"We're excited to introduce bots for the Messenger Platform," said Facebook exec David Marcus in a blog post. "Bots can provide anything from automated subscription content like weather and traffic updates, to customized communications like receipts, shipping notifications, and live automated messages all by interacting directly with the people who want to get them."

The three main capabilities within bots for Messenger listed by the company are:

• Send/Receive API. This new capability includes the ability to send and receive text, images and rich bubbles with CTAs [calls to action].
• Generic Message Templates. We think people prefer to tap buttons and see beautiful images, rather than learn a new programming language to interact with your bot. That's why we've built structured messages with call to actions, horizontal scroll, URLs and postbacks.
• Welcome screen + Null state CTAs. Our first principle was giving developers space to own the experience. Think of the message thread as your app. We're giving you the real estate and the tools to customize your experience. This starts with the welcome screen. People discover our featured bots and enter the conversation. Then, they see your brand, your Messenger greeting, and a call to action to 'Get Started'.

To help developers leverage these capabilities, Facebook is providing the Wit.ai Bot Engine to provide "ongoing training of bots using sample conversations. This enables you to create conversational bots that can automatically chat with users. The wit.ai Bot Engine effectively turns natural language into structured data as a simple way to manage context and drive conversations based on your business or app's goals."

To help get their efforts noticed, Facebook said developers can use Web plug-ins, Messenger Codes (scanning a provided code to start a conversation), Messenger Links and Messenger Usernames (both of which provide personalized links that can be shared anywhere online), which are detailed here.

Other Players
Microsoft and Facebook are helping to lead the way in bot development, but they aren't the only major players in this wide-open new game. There's IBM's Watson, of course, and Apple's Siri. And there's plenty more going in in this space. Chatbots.org publishes a "List of all chatbots (virtual assistants, chat bot, conversational agents, virtual agents) in the world" and a "New to Chatbot programming" forum site.

Pandorabots Inc. offers Artificial Intelligence as a Service (AIaaS), described as "a RESTful API for integrating artificially intelligent chatbots into applications," along with the Pandorabots Playground, "a free, integrated development environment for building chatbots loaded with features and tools."

For more on chatbot development, check out Why Developers Love Chatbots and Bots: An introduction for developers. There's also The Book of AI, describing "How To Build Chat Bots with The Personality Forge," which is "an advanced artificial intelligence platform for creating chat bots."

Still in the Early Stages
Despite the aforementioned uptick in interest, Forrester Research analyst Julie Ask earlier this month provided a word of warning, listing "a few big hurdles standing in the way of bots becoming the next big thing in 2016":

• Bot services must be convenient. Consumers want their mobile moments addressed by a better (read: faster and more contextual) experience. The apps that consumers use today like Uber, Yelp, Amazon or Facebook Notify, are amazing. There is nearly zero friction for the consumer to get what he or she wants immediately in context -- or what Forrester refers to as a mobile moment.
• The enabling technology isn't sophisticated. Great bot services will depend on artificial intelligence and insights. Consumers don't want to chat with machines that ask them dumb questions. Only one in four enterprises surveyed by Forrester even use location data to make mobile services more relevant -- let alone insights built on the immense context available.
• Developer tools are nascent. Fortunately today's chat platforms were built with extensibility in mind. That said, enterprise interactions demand enterprise quality -- major brands will not go to market based on a few developers trying this out in the office. Microsoft announced a Bot framework at Build 2016, and there will be many more to follow.

However, like most in the development community, Ask is generally optimistic about bots making a major impact in our lives and the lives of developers.

"Bots, and the chat platforms they run on, provide an amazing opportunity for brands to deliver contextual experiences on borrowed mobile moments," Ask said. "This is the first step on a journey towards a bright future where consumers no longer orchestrate their needs through content and services, but sit back and let the technology work for them."

Any experience with developing AI bots and chatbots? Comment here or drop me a line

Posted by David Ramel on April 22, 20160 comments

Share

• LinkedIn
• Facebook
• Twitter

Here Come the Coding Robots

Maybe this whole "teach everyone to code" thing wasn't such a good idea after all.

Emerging into the programming landscape is a new class of AI-assisted coding robots (or programming copilots) to train developers and augment their efforts. Their functionality is limited now, but they could signal the vanguard of an insidious uprising that one day will see you training your robotic replacement as your boss impatiently awaits his first command: "Siri (or whatever), wire me up a back-end to our new shopping app."

Related to the premise of this recent article questioning if data scientists will be automated and unemployed by 2025 ("Are we really just grist for the AI mill? Will robots replace us?"), they could be coming after YOU!

Or they could just be handy time-savers.

So say hello to your new little friends: Kite and Root.

Kite is described as "your programming copilot." It "augments your coding environment with all the Internet's programming knowledge." It's a desktop app that integrates with your tooling, fixing simple errors, providing example code and helping you search for answers to your problems.

Root is "a new educational robot," teamed up with an "accompanying programming environment, Square, to engage students with coding from an early age, retain their interest, and guide the growth of their coding skills."

The insouciant Kite ("No nerds were harmed in the making of this product.") is the product of a classic Silicon Valley start-up headquartered in San Francisco, with a Web site showing 13 team members.

"Kite is an artificial pair programmer," the requisite FAQ states. "While you're coding, it shows you examples and documentation for the libraries and terminal commands you use."

It so far has limited functionality, supporting only Python, running only on Mac OS X and providing open source plug-ins integrating only with Sublime Text, emacs, vim, PyCharm and Atom code editors. More languages, along with Windows and Linux support, are planned. Here's a video:

"As you type, we send your code to our servers as a query," the FAQ explains. "Our back-end analyzes your code and generates a response by querying it against terabytes of data, i.e., all the source code publicly available on the Web. This index is simply too large to ship with each client."

The company takes pains to assuage resulting privacy concerns (which you just know will be hashed to death on Hacker News) with that working model. "We never share your code with anyone," it says. "If we ever introduce new features that involve sharing code, we'll explicitly ask you first. But we'll never do something ridiculous like silently start showing all of your code to the world." Oh, and no nerds will be harmed in the process.

Root comes from the Wyss Institute, with a more educational bent. "One of the main challenges in teaching K-16 coding is the difficulty in finding frameworks that span a wide age range and appeal to broad audiences," the project's Web site states. "Databases aren't something that excite most students, and some concepts of coding can be too abstract to engage young minds."

"The robot, Root, is designed to fit into every classroom," the site continues. "It operates on whiteboards by driving on the vertical surface magnetically and -- using dry-erase markers -- draws, erases, and responds to lines drawn on the board. This ability to interact with the robot simply by drawing opens up potential for a broad range of unique instructional activities and games." Here's a video:

The accompanying Square programming environment, however, can help even somewhat experienced programmers.

Square "provides a sliding framework in which students can program the robot at different levels of proficiency -- from a blocks-based graphical environment designed for beginners and early elementary school students, up to a traditional text-based environment for older students or intermediate coders -- with the unique ability to toggle and automatically translate a program from displaying in any of the multiple levels," said the project site. "In this way, it helps support the transition to more advanced concepts and coding processes. The interface meets students at their individual level and evolves with them as their skills develop."

Its creator, the Wyss Institute for Biologically Inspired Engineering, is far from a Silicon Valley start-up, originating from the hallowed halls of Harvard University. It seeks to discover nature's engineering principles and apply them to real-world materials and devices to revolutionize healthcare. From that mission, somehow, came Root, to teach kids how to code.

"Root requires no special set up to begin operating on the whiteboard at the front of the classroom," the site says. "Unlike structured Web sites that teach coding as an individual activity based on exercises with unique solutions, the classroom environment is ideal for using multiple Root robots at the same time, providing opportunities for peer-to-peer learning and more freeform creativity. Coding with robots is an inherently dynamic experience; robot, environment, and students all react to each other, stimulating students to come up with diverse solutions and challenges together. Furthermore, robots have been shown to have a powerful appeal to all ages and demographics, making interactions and relationships with them far more compelling than images confined to a computer screen."

According to a write-up in the Harvard Gazette, "The Wyss team is currently seeking the right partners to help put Root into action in a classroom. The next step, said [Zivthan] Dubrovsky, is to develop a curriculum, including a coding module, around Root."

Over at Hacker News, Kite co-founder Adam Smith provided thoughts on the future of the "artificial pair programmer" in response to a comment about the need for an on-premises installation (you know, that whole HN privacy thing). A commenter said: "I was really excited, then very disappointed to see the code I type is sent to your servers. This immediately prevents it from being used by many businesses, including my employers. Super bummer :( Please get the on-prem sorted out pronto!"

Smith's reply:

We want you to be able to use it just as much! : )

There will be an adoption curve from early -> late adopters. Working through this is something we're committed to, and we know it will take time.

We're committed because we think it will be incredibly valuable to our field. We just can't imagine a future 50 years from now where programmers don't benefit from a smart back-end helping them work.

So we start with step one today, and here we go : )

And don't worry: No nerds will be harmed. (Yeah, right. That's probably what they said when they introduced Skynet and the Terminator).

How can humanity protect itself? Comment here or drop me a line.

Posted by David Ramel on April 19, 20160 comments

Share

• LinkedIn
• Facebook
• Twitter

Easier React Native (Android) Debugging

For this Android on Windows hobbyist, React Native remains an alluring but elusive shiny new toy to play with.

That's simply because creator Facebook has an iOS-first approach:

As React Native on iOS requires a Mac and most of the engineers at Facebook and contributors use Macs, support for OS X is a top priority. However, we would like to support developers using Linux and Windows too. We believe we'll get the best Linux and Windows support from people using these operating systems on a daily basis.

Therefore, Linux and Windows support for the development environment is an ongoing community responsibility.

They go on to say, "As of version 0.14 Android development with React Native is mostly possible on Linux and Windows."

Android/Windows 'Mostly Possible'
Just about a year after React Native was open sourced, "mostly possible" is close enough, I guess, as there's no guarantee the community can ever match core Facebook engineers' efforts on the iOS side.

And those efforts have certainly paid off. In last week's one-year review of the wildly popular project (30,000-plus stars on GitHub), Facebook's React Native team -- which has basically doubled in size to about 20 engineers -- explained how the year's effort has been "an epic ride."

So, after boning up on the basic React.js Web technology from which React Native was born, I did some basic tutorials. One thing I noticed on the Facebook site is that debugging information is a little sketchy, especially on the Android side. In the React Native docs, most of the debugging guidance consists of skimpy details about "Android logging" and using Chrome Developer Tools.

Too Many Moving Parts, Glitches
Following the provided Rotten Tomatoes movie tutorial, I ended up with a code editor opened up, along with one Command Prompt terminal window open to start the JavaScript packager, another Command Prompt opened to kick the debugging project off with a "react-native run-android" command, an emulator (I used Genymotion) or a device, and then a browser window opened up to localhost to debug the app using Chrome DevTools. That seemed like a lot of moving parts, entailing a lot of jumping around from window to another.

Also, perhaps because of my inexperience in using the DevTools, I ran into several glitches. Sometimes variables couldn't be evaluated at a breakpoint, though they were certainly in scope.

I tried "evaluate in console" on a variable and got an Uncaught ReferenceError ... variable "not defined" message, only to have it kick in moments later, for some reason (not sure what I clicked). I got a "This script is blackboxed in debugger" message in trying to set a breakpoint, which is apparently problematic to others, too. The iOS-first guidance (with repeated directions to use the Mac Command button, for example), didn't help, either.

I figured there had to be a better way, using one IDE or code editor to simplify things.

Looking to Quora for Editor Advice
Poking around the Web, I found a Quora question about the best editor with which to write React and React Native code. A prominent answerer pointed out Atom, Nuclide and the newcomer, Visual Studio Code.

"Nuclide is Atom enhanced by Facebook for React development," the answer stated. But it doesn't play nice with Windows: "There are two supported platforms for Nuclide, Linux and Mac OS X. Nuclide can be installed on Windows, but it is not fully supported. ... Some features of Nuclide may work on Windows, but the full Nuclide experience is not yet supported."

Worse yet: "Debugging React Native for Android is currently not supported except for the simulator logs."

That rules out Nuclide, though I had installed it earlier and again played around with it a bit just to prove it indeed wouldn't debug an Android app.

I uninstalled Nuclide, to try just Atom with a React.js package, for JSX support, which I had played around with earlier. I ended up wasting a lot of time just getting a project directory tree view to show up, again, apparently a known issue.

I also didn't have access to the React Native commands from the Command Palette, to start the packager, debugger, etc. So I would have to have the terminal windows open again.

Visual Studio Code 'Just Works'
So it was on to Visual Studio Code.

Guess what? It just worked!

Having just hit version 1.0, Visual Studio Code is Microsoft's lightweight, free, open source, cross-platform little brother to the outstanding Visual Studio IDE. It, too, has Atom connections, being based on Electron, which used to be called Atom Shell, a part of the Electron project.

Even better, it can leverage the outstanding React Native Tools extension. "This extension provides a development environment for React Native projects," the project says. "You can debug your code, quickly run react-native commands from the command palette and use IntelliSense to browse objects, functions and parameters for React Native APIs."

It's all true. I just hit Ctrl+Shift+P to open up the Command Palette, wherein I could start the packager (React Native: Start Packager) and kick of the project (React Native: Run Android). No more multiple Command Prompt windows. Breakpoints were set with single-click ease, unlike my experience with Chrome DevTools. Single clicks also let you choose your debug environment and your target (iOS or Android). Then a single click of the Start button and it fired up the packager and launched the debugger all in one motion. And it worked flawlessly with the Genymotion simulator. And the directory tree view just shows up.

A Couple Glitches
I did run into some minor glitches. Again, these probably stemmed from inexperience with the tool, but I'll detail them here in case somebody with the same programming aptitude I have is Web searching about the same issues.

In some Visual Studio Blog documentation, it says to hit Ctrl+P to invoke the Command Palette to install the React Native Tools extension.

Which is fine, but in some React Native Tools guidance, it says, in the Command Palette, to type "React Native" to choose a command from the drop-down list that appears, to Start Packager or Run Android, for example. I tried that and saw no commands. Finally I noticed that the screenshot shows a ">" character before the example command.

I thought that ">" symbol just indicated a prompt, but it actually shows up when you open up the real Command Palette with Ctrl+Shift+P. That's opposed to the aforementioned guidance that said to open the Command Palette with Ctrl+P. It turns out that Ctrl+P command is actually a "Quick Open" command, to bring up a prompt to open a file or symbol or do myriad other things. To run a command, you have to type the ">" character first after opening up the Ctrl+P prompt, and then you can run the React Native commands (or just open up the "real" prompt that shows the ">" symbol with Ctrl+Shift+P). A small thing, but it caused me some confusion.

I experienced another issue with debugging. Upon pressing Start to launch a debugging session on the emulator, I found the session wasn't stopping on the breakpoints I had set. The VS Code documentation says this will happen if the JavaScript packager is started outside the VS Code environment (as in a Command Prompt window), but that wasn't the case here.

Select 'Debug with Chrome' to Debug with VS Code
Mystified, I fooled around with various things trying to get it to work. From the emulator, I finally hit the Menu button to bring up developer options and selected "Debug with Chrome" to see if that would bring up a localhost browser or what. To my surprise, the debugging in VS Code kicked in -- the session stopped at the breakpoints, and variables could be inspected in a variable window, a watch window or by hovering over the variable name. Everything worked.

Ironically, I found an explanation for this behavior in guidance for Nuclide. "From the simulator, you will want to enable debugging the application. Press cmd-D (ctrl-D on Linux)," it says. "This will bring up the debug options for your application. Choose Debug in Chrome." It continues: "Debug in Chrome in this case is a bit of a misnomer since you will actually be debugging through the Nuclide debugger (based on Chrome Dev Tools)."

So that was it. You click on "Debug in Chrome" from the simulator to actually "Debug in VS Code." Go figure.

With those minor glitches overcome, the debugging experience with my sample Android app from within VS Code worked fine. Sure, Android/Windows support in React Native is lagging behind iOS and might be forever, but it can be done.

"In the year since the iOS launch, there have been quite a few apps released to the App Store, and high-quality Android apps are starting to appear too," Facebook affirmed in its one-year review.

Community Moves Project Forward
Facebook credited community involvement with helping to move the React Native project forward. "The adoption of React Native and its developer community have both grown much faster than we had expected," the team said. "Over 650 people have committed code to the React Native codebase. Out of the 5,800 commits to the codebase, 30 percent were made by contributors who don't work at Facebook. In February 2016, for the first time, more than 50 percent of the commits came from external contributors. With so many people from the community contributing to React Native, we've seen as many as 266 new pull requests per month (up to 10 pull requests per day). Many of these are high-quality and implement widely used features."

If I continue to learn in my blundering, hobbyist manner, maybe I can eventually contribute to some code documentation fixes or something to help the cause. In the meantime, I encourage those more proficient to contribute on the Windows/Android side of things. It definitely needs the help.

And I'm not alone in that opinion. Some recent commenters on Hacker News agreed with me, with thoughts such as: "It seems that React native on iOS is more stable than it is on Android. I wouldn't use it in production for Android"; and "When trying it for Android it didn't feel like a help at all. I still had to set up all this mobile dev environment boilerplate...."

And the No. 1 feature request on the ProductPains site for soliciting feedback reads: "Get Android to feature parity with iOS."

"Developing an app with React Native on iOS is a very good experience," the ProductPains post reads. "On Android (which I understand is still very new), everything feels a little more complicated and I find myself writing a lot more native code than I would like." So here's hoping for a little Android catch-up.

Have you tried creating Android apps on Windows with React Native? Please comment here or drop me a line.

Posted by David Ramel on April 18, 20160 comments

Share

• LinkedIn
• Facebook
• Twitter

'Why I'm Leaving [Insert Tech of Choice]'

I'm tired of writing about all the cool new things happening in software development, so I'm switching gears to explore the dark side things for a change. I'll take a look at the reasons people gave up on a technology, as taken from their ever-more-prevalent "Why I'm leaving ... " posts.

I was inspired to do this after reading comments in a Hacker News post about some apoplectic, profane purveyor of cartoons or something who let everyone know why he's leaving the Medium blogging platform.

So, instead of funneling expert advice as to why you should check out a technology (or organization or community), here are real-world examples detailing why people binned it. Who knows? It might be just as useful in making your technology decisions -- or wasting time on a slow Friday.

'Why I'm Leaving the R Community'
That was actually the title of a Hacker News post pointing to a post titled "Down and out in statistical computing."

This person provided a ready-made TL;DR (we used to call these "summaries") in a February post:

TL;DR: I'm resigning from all of my responsibilities and positions in the wider R community and ceasing development of new R packages. I no longer have faith that the R Foundation is capable of or interested in creating a safe and welcoming space for statisticians, scientists and programmers, and do not wish to offer support to an infrastructure lacking these capabilities and interests.

'Why I'm Leaving Ember'
This was posted to the Ember.js Discussion Forum in September 2014 and actually provoked a huge list of 147 comments (prepare to scroll if you're checking this baby out):

At first Ember seemed to be a great solution for development. But as a beginner in development (1,5 years) I felt more and more lost in the world of Ember. Ember and Ember-Data seem to be very powerful. But it is hard to get the whole picture in how things work together. The documentation shows many details of Ember, but not how they work together.

Multiple times I was stuck cause things didn't go as expected. The good thing was though, that Ember has a great community of very helpful and friendly people. So I went on the last months. But I've now realised, that Ember doesn't do the things the way it would seem obvious to me.

I've started with a full-stack framework (not sure if I should name it here) and in about 2 weeks I'm at a point which took me over 4 months with Ember.

'Why I'm Leaving Rails'
This was a February 2014 blog post on akonwi.github.io by Akonwi Ngoh, a software engineer at ThoughtWorks. He has contributed to Atom, React Native and other projects. The first bits read:

I gave up Ruby on Rails pretty quickly after I put my first app, intrnhuntr.com, into production. But recently, I've been reading about other developer's complaints about the Rails framework and the Rails way. One of the arguments being brought up is that the framework is "too big" and not for beginners. Here is pretty good blog post that identifies how Ruby on Rails is too complicated [Ed note: the link in original post was deleted because it doesn't work now].

Those posts aren't the main reason I'm leaving Rails but they are relevant and inspired this post. The main reason I'm leaving Rails is because it doesn't make me feel like a productive programmer. I've been using Node.js and CoffeeScript lately and everything is smooth. There is no scaffolding and somehow having a full-blown app generated that I need to muck around and change. Sure, you can remove what you don't need/want but why muck around when I can build what I need from scratch and keep it lightweight and efficient?

'Why I'm Leaving Unity'
This one, from a Unity3D forum post in September 2012, garnered nearly 50 comments of its own. Note that, almost four years on, some of these problems are likely ancient history:

Unity is very close to being a wonderful piece of software. However, my experience with it has been very painful.

Over the past month, I have been writing a voxel terrain system. Unity made this extremely easy to pull off, and I could get good results very fast. However, soon my problems began to set in. Over the course of about 2 weeks, around 15 different bugs occurred in the Unity engine, building up to a major show-stopping bug which caused Unity to crash repeatedly.

So basically, there are way too many big bugs for Unity to be of any real use. It's very very close to being great, and I'm sure if you don't have any problems, then it is fantastic. But unfortunately, I was not that fortunate.

'Why I'm Leaving Ubuntu for Debian'
Micah Lee posted this missive on his own site in January 2013. It freaked me out because I thought the guy's photo was in the top-right corner, and suddenly he took a sip of coffee or something while I was reading. Whoa! It was actually a looping video in which he mostly just stares at you. Ain't technology wonderful?

Cherry-picking parts of the post:

A lot of Ubuntu's recent decisions have been turning me off. It started a couple years ago when they changed the default desktop environment from GNOME to Unity. I had played with Unity when it was called "Ubuntu Netbook Remix" and I thought it was a fun toy, and might be easier to use on a touchscreen device than GNOME. But they made it the default before it was ready.

Another thing that started to annoy me was Ubuntu One, their cloud service. I could immediately see that if I were to use and rely on Ubuntu One, I would be locked in. It would be like needing iCloud or something, and I didn't like that idea. I was also noticing that my OS was starting to want me to buy stuff. Cloud storage space, music from the Ubuntu One music store straight from Rhythmbox.

Around the same time came the Ubuntu Software Center. At first I was impressed. They managed to make something way more usable than Synaptic Package Manager for finding programs to install. It wasn't always clear which packages were programs themselves or just dependencies of other programs, but Ubuntu added ratings, reviews, screenshots, and a nicer interface. But then they started integrating the Ubuntu Software Center with Ubuntu One. Then they started selling proprietary software through it. Then they started calling programs "apps," and featuring them with big graphical banners.

'Why I'm Leaving Debian'
Ok, that's not the title of the post under examination, but it contrasts nicely with the previous entry, doesn't it? I actually wrote this one up in a blog post titled "What's Wrong at Debian?"

The actual post in question was titled "so long and thanks for all the fish" on the Debian mailing list. (btw, what's with the lack of capitalization these days? I corrected most of the over-lowercasedness, but left this alone to show you what a journalist and former copy editor has to put up with in these crazy times -- surely a further sign of humanity's overall collapse.)

Sorry, I ramble when visually assaulted by a moving photo. Anyway, this was from a well-respected open source contributor, Joey Hess, in November 2014:

It's become abundantly clear that this is no longer the project I originally joined in 1996. We've made some good things, and I wish everyone well, but I'm out.

Note that this also constitutes an orphaning as upstream of debhelper, alien, dpkg-repack, and debmirror.

I will be making final orphaning uploads of other packages that are not team maintained, over the next couple of days, as bandwidth allows.

If I have one regret from my 18 years in Debian, it's that when the Debian constitution was originally proposed, despite seeing it as dubious, I neglected to speak out against it. It's clear to me now that it's a toxic document, that has slowly but surely led Debian in very unhealthy directions.

'Why I'm Done with 'Why I'm Leaving' Posts'
You probably get this by now, right?

Why I'm Leaving This Post
Out of room. Here are some more for your reading pleasure:

• Why I Left the .NET Framework
• Why I'm Leaving Subversion for Git
• Why I'm Leaving SEO
• Why I'm Leaving Workspaces (& Why You Should Too)
• Why I'm Leaving CAFM Behind!

Why are you leaving? Comment here or drop me a line.

Posted by David Ramel on April 15, 20160 comments

Share

• LinkedIn
• Facebook
• Twitter

As IoT 'Goes Mainstream,' How Should Developers Prepare?

The Internet of Things (IoT) has arrived, according to a new research report. As it "goes mainstream," joining mobile, Big Data and the cloud as ripe areas for software developers looking for career growth, technical opportunity and big salaries, how should they prepare?

Read on for insights into the burgeoning technology and expert advice tailored specifically for IoT developers.

The Opportunity: 'Internet of Things Goes Mainstream, According to New Verizon Report'
That's the headline of a news release from Verizon, which commissioned the research by Oxford Economics. "With an addressable market that includes more than 150 million cars that are not currently connected, over 300 million utility meters, nearly 1 million vineyard acres and 45 million people sharing goods and services in the U.S. alone, the IoT is now mainstream," said Verizon in announcing its new report. The "State of the Market: Internet of Things 2016."

IoT's arrival presents a great opportunity for developers who want to get in on the action early and possibly effect tremendous changes in the way we live. "Forward-thinking business and public sector leaders, as well as consumers and developers, are turning to the IoT to address some of society's most pressing social, economic and business challenges," Verizon said.

The Tooling
For developers addressing those problems, tools of the trade are vital. Verizon pointed out that some of the biggest players in the industry have come out with IoT development platforms, including Apple's HomeKit, Google's Brillo, IBM's Watson, Cisco's Jasper and, yes, Verizon's own ThingSpace.

"These and numerous other platforms populating the IoT marketplace promise a secure infrastructure and a set of tools that enable developers to build IoT applications, then manage the vast array of connected devices and the huge volumes of data those endpoints generate," Verizon said.

"In a nutshell, the platforms work to jump-start the development and deployment of IoT apps by radically simplifying the process for building and managing applications," the report continued. "Platforms provide a one-stop shop for development tools, secure network connectivity and everything else needed to launch, scale and manage apps."

The maturity of complete tooling platforms eliminates the hodgepodge approach that previously challenged IoT developers who had to jump through hoops to integrate different channels, processes and dev tools. Next we look at the needed skills to best utilize the new tooling.

The Skills
"Companies looking to make the most of IoT by bringing development in house are quickly realizing that they don't have the requisite skills they need to support it," ADTmag was told by Ryan Johnson, director of categories at Upwork, which provides a developer platform matching up freelancers with employers who post jobs. "Finding these people is proving to be a big challenge for companies."

Upwork recently conducted a little research of its own, coming up with the 10 most-wanted IoT developer skills, as measured by the percentage increase in annual job postings.

Skill	Increase
Circuit Design	231%
Microcontroller Programming	225%
AutoCAD	216%
Machine Learning	199%
Security Infrastructure	194%
Big Data	183%
Electrical Engineering	159%
Security Engineering	124%
Node.js	86%
GPS Development	66%

As you can see, some of these aren't your ordinary development skills. When was the last time you heard a hue and cry for more AutoCAD developers?

We asked IoT guru John Cohn at IBM if he was surprised by this list. If anyone can speak with authority on IoT development, it's Cohn, with the pedigree of being an IBM Fellow, MIT alumni and "Distinguished Agitator."

"A developer at heart and a modern day 'mad scientist,' John is constantly searching for the next exciting technology revolution, which puts him in lock step with today's new breed of IoT developers," said an IBM spokesperson.

"This data makes total sense to me because it's about 'things' and the fact that to be successful at developing them we have to bring a real world object together with real world code," Cohn said of the Upwork research. "Hands-on experimentation and testing between these two worlds is the only and best way to know if what we are developing will actually work and for every application to reach its full potential.

"This is why you see the advent of maker spaces. Much like open source communities, these are places where anyone can go to figure out how to make the physical thing that you are envisioning and bringing your code to life."

We asked Cohn what other dev skills are most in-demand for IoT development. "To be successful in the world of IoT, it helps to have a strong sense of design," he said. "IoT objects interact with people and need to be designed with the purpose, aesthetic and function in mind. An object that performs seamlessly and as expected becomes an indispensable 'enchanted object' -- to quote David Rose from MIT: 'an object that is difficult to use ... or does not behave as a user expects is in a bin somewhere two weeks after purchase.' "

For another take on hot IT skills associated with the IoT, ITCareerFinder listed the following in a post earlier this year:

• Business Intelligence
• Information Security
• UI/UX Design
• Mobile Development
• Hardware Engineering
• Networking

"The IoT is one of the most revolutionary technology trends of our lifetime -- and it's poised to explode," the post said. "These skill sets will maximize your salary and marketability in the proliferating Internet of Things."

The Interesting Case of Node.js
You may have noticed one of the more "traditional" skills listed by Upwork is Node.js, clocking in at No. 9 on the top 10 list. That JavaScript framework seems to be especially linked with IoT development, judging from a brand-new Node.js developer survey, which described the technology as "the engine that drives IoT."

"JavaScript and Node.js have risen to be the language and platform of choice for IoT as both are suited for data intensive environments that require parallel programming without disruption," the Node.js Foundation said in announcing the survey yesterday. "JavaScript, including Node.js and frameworks, such as React, have become the de facto choice of developers working in these connected, device-driven environments with 96 percent of IoT respondents indicating they use JavaScript/Node.js for development."

Indeed, IoT has its own "developer type" in the new report, along with the traditional categories of "front-end" and "back-end" developers.

Here's a chart from the survey listing other programming languages, in addition to Node.js/JavaScript, used by those developer types:

"JavaScript and Node.js were popular among back-end, front-end, and IoT developers," the Node.js Foundation said. "Other languages, beyond JavaScript, that were popular for all three developer types included PHP, Python and Java. However, when looking to the future, back end, front end and IoT developers planned to decrease their use of Java, .NET and PHP (PHP averages a 15 percent decrease) and increase the use of Python and C++."

The Rewards
A lead IoT engineer can command a $120,000 annual salary, according to careers site SimplyHired, which reported an average salary for "IoT engineer jobs" of $80,000.

Careers site Indeed reports 1,740 "Internet of Things" postings with salary estimates of more than $115,000 and 539 "IoT" postings with salary estimates of more than $130,000.

Of course, it's not always about the salary. Developers also weigh aspects like career growth opportunity, technical opportunity, challenges and more. We asked Cohn to opine on the benefits of developing in this growing technology space.

"One of the things that I find most interesting and rewarding about developing for IoT is the you have the ability to interact and move back and forth between the digital and the physical," Cohn said. "Almost anything is possible when you have this convergence of low cost, low power embeddable electronics, ubiquitous connectivity and cloud computing. With that you get access to essentially infinite computing in every device no matter how tiny. Code is abstract, but once you connect it to devices through the cloud, it becomes alive and you are developing things that make a real difference in the world. There's never been a time in technology when you can infuse this much intelligence into any device and that's exciting."

How To Prepare
We also asked Cohn how developers should prepare for the IoT -- what skills and technologies they should be focused on.

"The technology around IoT is evolving quickly and with so much light weight, agile development going on, you have to be able to move from one to the next," Cohn told ADTmag in an e-mail. "Developers now must be hip to all of the changes in programming languages and evolving hardware platforms. They have to have their finger on the pulse of open source communities and the tools and methodologies coming out of them.

"You have to be hands on with IoT because developing from an abstract point of view no longer works. Creativity and innovation are spawned from an understanding of what a device can and can't do. Which means that you have to write the code and then physically deploy the devices yourselves -- again, bridging that gap between the digital and physical worlds."

Bridging that gap is going to be an interesting challenge, as research firm Gartner last fall predicted some 6.4 billion "things" would be connected this year, a 30 percent increase from last year. Looking further forward, Juniper Research sees that number climbing to an astounding 38 billion units by 2020.

Furthermore, Upwork noted that Gartner identified insufficient staffing/expertise as the top-cited barrier for organizations currently looking to implement IoT in a February report.

"Get in now and start building things yourself," Cohn advised, continuing to emphasize his hands-on philosophy. "IoT is not a spectator sport, it's fast-moving and fun. You can be making cool things to enrich your life in a matter of hours ... you may even create the next 'must have' device. Like most things, the only way to learn it is to practice it."

Posted by David Ramel on April 13, 20160 comments

Share

• LinkedIn
• Facebook
• Twitter

Comparing React.js Tutorials

I immediately sensed React Native was "game-changing" technology upon first hearing about it, and have been eagerly waiting to try it out. Unfortunately, I've had to wait quite a while because React Native creator Facebook has a definite iOS-first bias, and I don't do any Apple stuff. So I had to wait for Android/Windows functionality to catch up.

It still hasn't, and possibly never will, as Facebook has admitted its engineers are into Macs -- their primary dev machines of choice -- and they're relying upon community outsiders to move the counterpart Android/Windows effort forward. That doesn't seem to be happening very quickly ("As of version 0.14 Android development with React native is mostly possible on Linux and Windows."). "Mostly possible" just doesn't inspire a lot of confidence.

But I dove in anyway. I managed to get through the basic Facebook tutorial and some others and got some React Native apps running, but it was quite a challenging experience and I didn't feel comfortable. I thought boning up on the original React.js technology for Web development -- which led to React Native -- would give me a better understanding. I worked through the occasional independent tutorial and managed to create a few React.js Web sites. Still didn't feel comfortable (I've never really liked JavaScript -- I prefer more orderly classes in the object-oriented approach). So I thought working through some real online React.js courses from start to finish would give me a more solid, soup-to-nuts foundation.

Apparently I'm not alone. Just yesterday, a Hacker News post asked: "What are some good React tutorials?"

For those who might be in the same boat and wondering where to start, here is my comparison of two courses I just finished: React.js Fundamentals by Tyler McGinnis, and Learn React.js: The Basics, on Lynda.com.

The Cost
First off, the basics. React.js Fundamentals is a free course, offered along with another course (ES6/7 for React, $45) and some upcoming paid courses in McGinnis' React.js Program. The Learn React.js: The Basics course comes with a Lynda.com subscription. For brevity, I'll just refer to McGinnis or Lynda.com from here on in discussing the two courses.

Prerequisites -- the Target Audiences
"A solid understanding of JavaScript will help but isn't entirely required (you might have to do some extra research though)," says the McGinnis tutorial. "If you have no JavaScript experience but you're coming from another programming language, you'll be fine."

"Up and Running with React.js is not an intro level course," says the Lynda.com tutorial (apparently they changed the name of the tutorial somewhere along the way). "I'm going to assume that you're taking this course with some skills already in place. Number one, you should be familiar with HTML5 and CSS3. We'll also be doing a bit of work with Bootstrap. Also, React.js is built out of JavaScript, so you need to know some JavaScript."

The Approaches
Lynda.com, part of a far more established and complete overall offering, just uses videos, though a nifty transcript is provided just below each video, with text passages highlighted as the corresponding video rolls along. Together, they add up to 1:32 of tutelage. Along with the videos, you can download an extensive code package and tinker with code in an IDE. I followed instructor Eve Porcello's model and used Sublime Text for my IDE. The code is divided into chapters corresponding to the videos, with "start" and "finish" folders for each chapter that provide JavaScript and other files as they appear at the beginning of a video and how they appear at the end, reflecting the changes made by Porcello.

McGinnis uses a different approach, starting out with text to generally introduce a concept and following up with a video (total duration of the videos is about 2.5 hours, approximately an hour longer than Lynda.com) to work through lessons on the concepts just discussed.

"The whole premise of React.js Program is that the student is going to have three different learning approaches thrown at them," McGinnis explains in an introductory note written at 3:30 a.m. "First via text. Second via a video. Third via curriculum."

The curriculum comes in project files for each section hosted on GitHub. At the beginning of each video, McGinnis points to the corresponding code branch. I simply downloaded the overall project structure, opened up the base folder in Sublime Text and, with each new lesson, scraped the new code and pasted into index.js or whatever files were appropriate.

What You Build
Lynda.com runs through the creation of a bulletin board app, in which you can create notes, edit them, delete them and move them around on a virtual "board."

McGinnis' project is called GitHub Battle. It lets you enter two GitHub usernames and pops up a comparison listing personal details along with number of followers, number of developers they're following, public repos and more, along with photos of the two "combatants."

Technologies Used
Both courses utilize NPM and Babel.

With McGinnis, after the introduction but before you even start to code, you get lessons on set-up first, using NPM, Babel and Webpack. You learn how to download and wire everything up first, so upon making your first component, you can view it on localhost and immediately see coding changes reflected on the "Web site."

With Lynda.com, you dive into simple code sooner, after learning about things like Chrome development tools and Babel, the latter used to convert JSX code into plain JavaScript. However, to avoid Chrome's CORS security issues, Porcello opens up her index.html file in a Firefox browser to demonstrate the incremental functionality. It's not until the last chapter when you learn how to create a project with NPM and see it running on localhost via the lightweight httpster server.

Course Curricula
Here's a comparison of the respective curricula, with the Lynda.com curriculum abbreviated and condensed for space:

McGinnis	Lynda.com
Intro to the React Ecosystem	What is React.js and why is it so fast?
Setting up your first React component with NPM, Babel, and Webpack	Setting up React tools with Chrome
Pure Functions. f(d)=v. Props and Nesting Components	React.js syntax
this.props.children and getting started with React Router	Introducing JSX and Babel
Container vs Presentational Components, PropTypes, and Stateless Functional Components	Creating a React component, using properties, handling events
Life Cycle Events and Conditional Rendering	Adding state to the note component
Axios, Promises, and the Github API	Using refs and PropTypes
Rendering UI	Adding multiple child elements
More Container vs Presentational Components	Updating, removing and adding notes
Private Functional Stateless Components	A deeper look at keys
Building a Highly Reusable React Component	Component life cycle, mounting components
React Router Transition Animation and Webpack's CSS Loader	Setting properties, updating components
Further Curriculum: a Weather App	Integrating React with a server, hosting with npm and httpster
Favorite Resources	Enhancing React with additional frameworks

The Subjective Stuff
I found both courses useful, and I found them even more useful taken together. The different technologies and approaches provide insights you wouldn't get by just following one course (which is probably true for learning any technology).

If I had to choose, I'd go with the young upstart, McGinnis. Part of the reason is the focus on retrieving real live data, in this case from GitHub users. Data is king, and I'm interested in presenting it in new and useful ways, especially tapping into public data sets. That can lead to advances in my own field -- such as data-driven journalistic techniques -- and probably just about any other endeavor.

But the main reason I preferred McGinnis is simple: the personal touch provided by Tyler McGinnis himself. The man appears to be a human dynamo, with more irons in the fire than I can count. He clearly understands the value of providing a human face to connect with, offering a little webcam image of himself to accompany the coding screen as he narrates a video. And he offers personalized touches throughout the course and superb support.

Here's his intro:

Hi! I'm Tyler. I'm currently the CTO of Spero where we're building a React Native app that is helping connect individuals affected by Cancer. Before Spero I found my love for teaching at DevMountain where I was the Lead Instructor and Curriculum Engineer. I've taught 1000's about React through both my personal blog and Egghead.io. I'm a Google Developer Expert. I organized React.js Utah and ReactWeek. Sometimes I ramble on JavaScript air as a Co-host and I also help run React Newsletter and Front End Newsletter.

In addition to those things, he's trying to grow his tutorial business, and he appeared to be available for help pretty much around the clock. He invited each course participant to the associated Slack channel, where he provides personalized help for seemingly hundreds of users.

Here's an example. When an npm module naming squabble caused a disgruntled developer to "unpublish" his npm packages, it caused thousands of JavaScript builds around the world to break. I experienced this problem live, as it happened, while working through the McGinnis tutorial. I couldn't figure out what went wrong until I finally logged into the Slack channel, where I saw others were reporting the same problem. McGinnis was online, of course, and quickly pinpointed the problem and provided updates, saying that he expected it to be fixed within the hour, which it was.

I was duly impressed, and apparently others are, too. His tutorial came in for some kudos on the aforementioned Hacker News post.

"I am using this one and like it very much so far," one commenter said.

"+1 for Tyler and his courses," read another post. "There's something great about the way he presents the knowledge I can't quite put words to. He manages to 'pull back the curtain' just enough to satisfy my curiosity without allowing me to go down a tangent path of little details that contextually don't matter until later. His frank demeanor instilled a lot of trust in me from the get-go, as well, which in turn left me satisfied when he says things like, 'that doesn't matter right now -- we'll get back to that when it's important.' Usually in tutorials or classes it's either high or low of that mark -- either I don't believe I will end up learning about some hidden principle that ends up being important, or I'll have way too much depth dumped on me and I'll be lost."

"I am working through Tyler's free tutorial and so far have enjoyed it," read yet another post.

The Lynda.com tutorial is no slouch either, having been viewed more than 12,000 times. Eve Porcello has a nice delivery, and besides React, the instructor at Moon Highway is fluent in Flux, JavaScript and Node.js. She also teaches courses on Engineer Onboarding, Agile, UX Planning and Design, and Project Management.

One advantage of the Lynda.com approach, if you're paying for a subscription, is access to the huge inventory of other courses. McGinnis, while offering his React.js for free possibly as an intro to his for-pay options, has only the "ES6/7 for React" course available now, as mentioned for $45. Others, including Redux + Immutable, React Native and Universal React, are all labeled "Coming Soon" for $245 each.

I don't see how his single-handed support scheme can scale as his business grows (how many 3:30 a.m. "nights" can you hack while holding down a day job?), but it will be interesting to watch.

More Tutorials
In the meantime, here's a list of some other tutorials mentioned in that aforementioned Hacker News post published yesterday to keep you occupied:

• Full-Stack Redux Tutorial (Teropa)
• Getting Started with Redux (Egghead.io)
• Facebook Tutorial
• Building Your First React.js App (on Medium.com)
• React for Beginners (Wes Boss)
• Learn Raw React — no JSX, no Flux, no ES6, no Webpack ... (James K. Nelson)
• Thinking in React (Pete Hunt, Facebook)
• Build with React (Pete Shen)
• Learn Webpack and React (SurviveJS book)
• Removing User Interface Complexity, or Why React is Awesome (James Long)
• ReactJS For Stupid People (Andrew Ray)
• React.js Introduction For People Who Know Just Enough jQuery To Get By (React for Designers)

What React tutorials have you found most useful. Comment here or drop me a line.

Posted by David Ramel on April 11, 20160 comments

Share

• LinkedIn
• Facebook
• Twitter

From Uber Code Puzzles to Secret Console Messages, Firms Scramble To Attract Dev Talent

It's a seller's market in the software development arena these days, with companies resorting to all kinds of new and creative ways to identify and attract hard-to-find talent in the face of a daunting skills shortage.

Case in point:

Just the other day I was working through a Web development tutorial and visited a site to inspect how it was using React.js components. I hit CTRL + SHIFT + J to open up the console to get a peek into the inner workings of the site, imgur.com, only to see this message:

"You opened the console! Know some code, do you? Want to work for one of the best startups around? http://imgur.com/jobs."

After submitting my application (LOL! JK, boss!), I investigated further. Even though this attempt at innovative recruitment was ridiculed on Reddit (Is that enough to be a webdev?), some readers pointed out that several sites are using such console messages.

And right after my experience, news came that rent-a-ride company Uber is using "Code on the Road" puzzle challenges from within its app to recruit developers.

"I got asked by @uber if I wanted to "code on the road". heard @google did something similar for recruiting, v smart," said one tweet on Twitter today, one of many reporting on the newly discovered recruiting device.

During the ride, the app reportedly pops up in certain targeted areas where developers are known to lurk.

"We are always looking for new ways to reach potential candidates that want to join our team and help us solve interesting problems," an Uber spokesperson told Business Insider. "If you're in a place where a lot of people work in tech, you may see our 'Code on the Road' challenge within the rider app. The option to play gives interested riders the opportunity to show us their skills in a fun and different way -- whether they code on the side or are pursuing a career as a developer."

The "something similar" that Google has reportedly done might have been a reference to a coding challenge that pops up when certain programming-related search terms are entered by a user. That was reported last summer by Max Rosett, who actually landed a job at Google through the unique mechanism.

"Foo.bar is a brilliant recruiting tactic," Rosett said in an online post titled Google Has a Secret Interview Process ... And It Landed Me a Job. "Google used it to identify me before I had even applied anywhere else, and they made me feel important while doing so. At the same time, they respected my privacy and didn’t reach out to me without explicitly requesting my information."

So, even though developers recently surveyed by Stack Overflow indicated that they were turned off by recruiters using the brain-teaser approach, coding challenges definitely work in certain situations.

In fact, a company called HackerRank is making great use of the approach, having earlier this year released a mobile app to extend its strategy of identifying top talent through competitive code challenges.

"The app lets job-seeking engineers request a coding challenge from a participating company with one click," we reported at the time. "They can solve the problem at their leisure and share their availability for an interview, after which they are guaranteed a response from the company within five days. If they don't pass muster, they're notified, so they know if they're advancing to the next round of the process or not."

Not that such coding challenges are brand-spanking new. A company called Quixey, founded in 2009, made great use of the approach. "Early on, Quixey built a coding challenge called Quixey Challenge to attract engineering talent," according to Wikipedia. "The contest asked participants to solve a bug in a famous line of code in under one minute to win $100 and a Quixey t-shirt. In December 2011, the contest yielded 38 winners, five of which became serious candidates for three open positions."

Of course, Big Data has also been applied to the task for a couple years now, as we reported in 2014. At the vanguard of this movement was a company called Gild Inc., which uses a "Gild Score" to rank devs. In addition to traditional factors such as the programming languages developers know, that score will be based on how active and helpful they are in software development social sites and how much open source code they publish -- and how good that code is.

A white paper by eQuest sheds more light on using Big Data in the general recruitment process.

And it's not just the software development industry that's resorting to new recruitment vectors, as Social-Hire.com listed creative approaches by such companies as Ikea. "The Swedish Giant cleverly concealed job descriptions inside every pack of furniture sold," the site said. "The campaign cost nothing -- customers literally delivered career information to themselves, and it resulted in 4,285 applications and 280 new hires."

As the skills shortage persists, expect new ways to come online to woo developers. As we reported earlier in an article about developer evangelists, traditional outreach messages such as advertising just don't cut it any more with that finicky bunch.

In what innovative ways have you been recruited for a development job? Share here in the comments or drop me a line.

Posted by David Ramel on March 28, 20160 comments

Share

• LinkedIn
• Facebook
• Twitter

npm Naming Squabble Breaks Thousands of JavaScript Builds

You know that sickening feeling when you're happily coding along and all of the sudden your project won't build and you just can't figure out why? That can lead to hours of nerve-wracking, stomach-churning troubleshooting.

Now imagine that scenario played out thousands of times over because of an npm module naming squabble that no amount of troubleshooting could remedy.

That's what happened last night when community contributor Azer Koçulu "un-published" a large number of his npm modules under threat of corporate lawyers (the lawyers ... always the lawyers). Because one JavaScript module named kik encroached upon the copyright of the company Kik, which produces a messaging app.

In the end, npm Inc. -- the organization overseeing the Node Package Manager -- took the "unprecedented" step of "un-un-publishing" one wildly popular module used in thousands of projects, to the relief of developers everywhere (except perhaps Koçulu).

That's all well and good and makes for good copy and plenty of sniping fodder on sites like Hacker News, and will lead to much agitated debate about proposed rules changes.

But I'm here to give you a taste of how things like this manifest themselves on the front lines and affect people just trying to get some work done -- even it the work in my case was just educational.

Here's how it all played out from my little corner of the dev world.

Last night, after spending all day writing about coding, I was trying to practice it, plodding through a tutorial on developing a React.js Web project.

I made a routine change involving some style formatting stuff, checked to see if the webpack dev server was running and refreshed the localhost page. Nothing. I couldn't figure it out. I took the styling out, and it worked again. I put it back in, and it broke again. I looked at this, I looked at that. Finally I logged into the Slack help channel the moderator had set up, looking to see if anyone else was running into the same thing.

They sure were.

The culprit was one of the "un-published" modules named left-pad (on GitHub here) for padding strings. NPM reports it was downloaded more than 2.5 million times in the last month.

"Im getting an odd error on npm install of the es6-for-react repo, has anyone else seen this?" asked one coder at 6:04 p.m. on the Slack channel.

"Interesting," replied Tyler, the course instructor, who immediately found and pointed to the same issue raised on GitHub that reported "npm gives me this error: No compatible version found: [email protected]."

"Left-pad gone up and bounced outta here," Tyler continued. "Not sure why... That issue was 12 minutes ago." He soon reported yet another GitHub issue: "npmjs.org tells me that left-pad is not available (404 page)."

"The world is burning," Tyler added.

There was more give and take with comments expressing thoughts like "wow" and "damn" and "wtff" and so on. You could just feel the tension and frustration mounting. Imagine that playing out all over the world, affecting many projects that developers depend on for money, not just for learning.

Thankfully, the angst was short-lived.

"It works now," reported the original dev who raised the issue, at 6:32 p.m.

We were soon enlightened as to the cause when someone pointed to a tweet by npm Inc. co-founder and CTO Laurie Voss. It read: "Hey npm users: left-pad 0.0.3 was unpublished, breaking LOTS of builds. To fix, we are un-un-publishing it at the request of the new owner."

Voss followed up. "Un-un-publishing is an unprecedented action that we're taking given the severity and widespread nature of breakage, and isn't done lightly."

That led to a lot of Twitter debate about the propriety of such a move.

"Even within npm we're not unanimous that this was the right call, but I cannot see hundreds of builds failing every second and not fix it," Voss tweeted, following up with, "This action puts the wider interests of the community of npm users at odds with the wishes of one author; we picked the needs of the many."

Koçulu explained his side of things in a Medium post titled "I've Just Liberated My Modules." Here's part of it:

When I started coding Kik, didn't know there is a company with same name. And I didn't want to let a company force me to change the name of it. After I refused them, they reached NPM's support emphasizing their lawyer power in every single e-mail CC'ing me. @izs [npm Inc. CEO] accepted to change the ownership of this module, without my permission.

This situation made me realize that NPM is someone's private land where corporate is more powerful than the people, and I do open source because, Power To The People.

Summary; NPM is no longer a place that I'll share my open source work at, so, I've just unpublished all my modules.

This is not a knee-jerk action. I love open source and believe that open source community will eventually create a truly free alternative for NPM.

I'm apologize from you if your stuff just got broken due to this. You can either point your dependency to repo directly (azer/dependency) or if you volunteer to take ownership of any module in my Github, I'll happily transfer the ownership.

[Note: After this article was originally published, Kik the company has posted their side of the story.]

This is going to shake up npm-land something fierce. I can't wait to see the promised "post mortem."

In the meantime, you can amuse yourself with the debates still raging on at Hacker News, where Koçulu's post has garnered 1,278 points and more than 660 comments. Of course, Reddit is also having a go at it. And Twitter is all over it.

Me? I ended up finishing the tutorial. No harm, no foul. But what if your livelihood depended on code that suddenly broke due to circumstances beyond your control, and the situation wasn't remedied so promptly?

Something to think about.

What do you think about this whole mess? What needs to be changed? Comment here or drop me a line.

Posted by David Ramel on March 23, 20160 comments

Share

• LinkedIn
• Facebook
• Twitter