John Tomizuka, co-founder of Taqtile, a leading mobile app development shop, gave a good keynote talk this morning. It was different in that it wasn't your typical “unicorns and rainbows everything is fantastic sort of keynote”, but more of a "here are some of our successes and what we learned and here are some of our failures and what learned" sort of talk. I thought it was a nice departure and showed that indeed, we all do mess things up, but if we are smart, we learn from those mistakes and find better ways to work, as a result. John talked about the importance of empathy and looking out for the best interest of your client and what it is that they really need, not what you or a few other people think would be cool. He also emphasized that you need to know your data so you can house it accordingly to keep it from becoming a bottleneck. Making sure application-critical data is accessible is another key aspect of a successful deployment and user experience.
John stressed the importance of finding and using the right tool for every job; many of the tools they use were discovered by the need to solve a problem. A few of those he spoke of were:
Finally, John talked about the importance of UI planning and wireframing and noted that wireframing doesn't have to be hard, all you really need to do is sketch out your designs. Simple steps like wireframing ideas and presenting them to your client early can really save time and money in the end.
Using WebWorkers and Processors to Bend Data to Your Will by Matt Priour
This was another talk that I went into not really knowing what to expect. The Web Workers specification defines an API for spawning background scripts in your web application, allowing you to do things like run computationally intensive processes in an isolated thread without blocking the UI or other scripts to handle user interactions. This means you can run your process in the background without it impacting the foreground; so you can still, for example, pan and zoom around your map. Web workers have no access to the DOM, so they cannot interactive with your UI and they have a very simple API. Other important points are that the script you pass into your Worker must from the same origin and the value in reusing workers, as creating and destroying them is costly. The Esri/workers/WorkerClient class is the primary entry point for interfacing with background workers.
Matt got to the good stuff and did a few demos. In the first demo, the application allowed him to upload a zipped shapefile and display it on the map. He chose a point shapefile containing 9000 records, and then buffered the points on-the-fly in about 15 seconds using background workers and the new geometry engine, all while being able to pan and zoom around the map.
Some other applications he demoed:
This last one is pretty interesting as it has around 50,000 well locations in the point layer.
Data Automation and Scripted Publishing using ArcGIS Solutions by Chris Buscaglia, Mike Miller, and Claire Price
ArcREST?!?! How have I not heard of this before today? ArcREST is a set of Python tools to assist working with ArcGIS REST API for ArcGIS Server, ArcGIS Online, and ArcGIS WebMap JSON. The readme says it's incomplete, but during the talk they said it covers just about everything that the REST API exposes. ArcREST should be pretty independent of ArcGIS versions but will definitely work at 10.2+. There are a few arcpy dependencies, but not many. Some examples of tasks you can do with ArcREST include:
- Use it in conjunction with arcpy.CreateReplica to pull down data from a hosted feature service into a local file geodatabase for backup.
- Query a featurelayer and pull it down as a feature class locally.
For Portal, you can create and add users to groups and create a backup of your data and export it to an FGDB (basically same as using the Export to FileGDB menu choice in Portal).
Chris discussed a set of tools they developed to help communities aggregate and update parcel data, called Community Parcels (https://github.com/Esri/community-parcels-python). Using ArcREST, this tool allows users to take a local dataset and push it up to a feature service, updating the feature service with the data from the local dataset, all using Python and the REST API. Pretty slick.
Finally, they discussed some reporting tools they developed for utilities that allow them to pull data from the REST API and use it to generate custom reports:
Using the File Geodatabase API by Lance Shipman
The file geodatabase API is a non-ArcObjects method to access the file geodatabase, no license required. It is a coarse-grained API written in C++. There is no support for pre-10.0 file geodatabases. It has an Apache license and Lance said that you can "...basically do anything with it...". Available for Win, Mac, Linux at varying levels.
All content in a file geodatabase can be opened for read access, with the exception of network indexes. Version 1.4 is in verification stages and coming soon, like next week.
There are gotchas on inserts, deletes, or updates to several dataset types. Rasters are not supported neither are attachments or joins.
In spite of all of those downer statements, you have to remember that this is a coarse-grained API that is NOT intended to take the place of using ArcObjects to access file geodatabases.
The good news for many of us is that a .NET wrapper has been available since version 1.1. When you download and unzip the API, there are some pretty good C# samples (C++ also) that can get you going. I was querying a local file geodatabase with a C# console app in about 15 minutes after making a few minor configuration changes to one of the samples.
The File Geodatabase API can be downloaded at http://www.esri.com/apps/products/download/
Python: Working with Scientific Data by Kevin Butler and Shaun Walbridge
This talk focused on using scipy and other scientific packages to manipulate and convert data into usable forms for GIS consumption. They also talked about the pandas library, which I have tinkered around with some myself. Kevin described pandas as "wickedly fast", which is pretty much true. The basic takeaway from this talk was that Esri is going to have more Python modules shipping with the standard installs and they will also have additional optional add-on installs of other packages available as well. This presentation is available at http://4326.us/scipy and is pretty packed with information.
Well I gotta say that I think the Smart Mapping engine Esri has created is the most significant cartographic capability they have done in years. I say this because I believe they have taken so much of the art and science (Tradecraft) of the true cartographers and have enabled those of us wannabe’s and even the ones that don’t even care and have taken all excuses away from making good looking and useful maps. What Smart Mapping does is that it automatically looks at your data/layer that you are trying to map and auto-analyzes it for a couple of things out of the gate. First it looks at the spatial distribution of your data and makes a first guess at spatial extent. So you don’t have to zoom in or out to get to an appropriate scale to look at what you have. Not a big deal but a click saver. Next, based on whatever attribute you select it makes a determination of the type of data it is and makes some prioritized suggestions on the type of symbology you might want, and what an appropriate color ramp will be, which is determined by the particular base map you are using and the data itself. After you pick an option you think might be nice you can further fiddle with the parameters, break points, size ranges, which classes to no-show, etc. Total flexibility once you have the selected the base option.
So what I think this is going to mean is that we are going to see two things occur. First, the general cartographic quality of maps coming from our community is going to jump a level – again they have embedded cartographic tradecraft into the tools for us all. Second, the usability of these tools is going to suck us all into deeper data exploration and we are just going to get smarter about the information we are looking at. Super easy tool to manipulate how data is displayed while the immediate response is going to draw us into the data and what it is saying – we will spend more time exploring.
So Smart Mapping is awesome, but right now, it is only available in AGOL Map Authoring. They say it is on the roadmap for ArcGIS Pro, and frankly that should be a priority. Why not put this capability in the handle of the most skilled analysts! Anyway, nice job Esri I can’t wait to get back home and start exploring my data and publishing cool maps.
The keynote presentation by John Tomizluka of Taqtile started off day 2 of the Esri Developer Summit. He entertained us with a review of all the interesting mobile app deployments his company has undertaken over the years. From deploying a streaming app for use in a location that ended up having no wireless service, to trying to push short term coupons to a million users – all hard lessons that have guided them into becoming a top mobile application development shop.
Integrating ArcGIS Data Reviewer and Workflow Manager for Server to Automate Data Quality Workflows
Jay Carey and Tope Bello
This was a session that I really enjoyed attending. The presenters provided a thorough overview of the Data Reviewer product and how to use it with Workflow Manager, as well as covering how to develop custom applications that pull out the information you may be interested in. The workflows can be run from the desktop clients ArcMap and ArcGIS Pro, or as scheduled tasks using the Data Reviewer for Server extension. There are 42 automated checks that can be run covering Attribute, Spatial, Feature Integrity, and Metadata validation. They provided a link to a downloadable poster covering these automated checks, but it just takes you to the standard product page. I will keep digging.
Data Automation and Scripted Publishing
Chris Buscaglia, Mike Miller, and Clare Price
Nice technical talk covering the ArcREST package, a set of python tools developed to assist working with the various ArcGIS REST APIs - Server, Portal, and ArcGIS Online. The tool package makes it easy to develop scripts for automating tasks like:
- Portal Content Backups
- Adding Fields to hosted Features
- Speeding up service installations
- Adding or removing data from sites
- Registering content
- Generating reports on user/group content
The package can be downloaded from the Esri ArcREST GitHub repository.
ArcGIS Web AppBuilder: Customizing and Extending
Tom Wason and Moxie Zhang
Walking into the packed large conference room, with folks standing along all the available walls, it was apparent that AppBuilder is a very hot topic at this year’s summit. The presenters provided an overview of the two components you develop with the application: Widgets and Themes, and then followed up by launching code demonstrations for each component using the Developer Edition. While there was not enough time to go into all the areas that could have been covered, they did a great job providing a starting point for anyone interested in using this tool. The Developer Edition can be downloaded for free at https://developers.arcgis.com/web-appbuilder/.
Plenary and Keynote John Tomizuka
Great true stories from the trenches of gigantic mobile app deployment. How will your app scale when Matt Lauer of the NBC Today Show announces it to the world? John can tell you how it went for him. John described the pitfalls and the tools he used to climb out of failures that ruined his Thanksgiving and Christmas as his team overcame the challenges of this app and a few others. Among the heroes were Heroku for smart scaling, Memcache for memory management, ElasticSearch for filtering big data, Applause for testing.
Predictive Analytics with IBM Watson Analytics, SPSS Modeler and Esri - Venky Rao & Armand Ruiz of IBM
Not sure how I wound up in this presentation, all I know is that showing up late for the UX talk late was a mistake – totally packed. I have never worked with SPSS, which is an IBM product for modeling and analysis, but was impressed with this drag and drop modeling interface similar to Esri’s model builder. I was impressed with the UI but impressed with the integration with Esri’s ArcGIS Online that IBM had only recently implemented within the last 2 months. The demo included an integration with Twitter that was able to publish tweet locations in ArcGIS Online.
Kelly introduced us to the Portal REST API and Matt Driscoll followed with demonstrations of Web AppBuilder customizations as well as a JS boilerplate available on GitHub. Kelly described how the Portal REST API can be accessed directly or through the Esri JS API to access data and web maps and pointed us to resources for authentication and accessing secure services. The Web AppBuilder has a couple new widgets, the time slider, and a map swipe tool. Matt began by talking about templates, which he described as a way for developers to create apps that can be easily configured by non-developers. He followed this with a demonstration of the Esri JS Boilerplate accessible from GitHub.
The main idea in this presentation was how to make your app load faster. The hero here was Require JS, which is a JS module loader that will “improve the speed and quality of your code”. These guys also discussed how Gulp or Grunt can be used to ‘watch’ your code as it's being written to catch errors.
Geo-enable Your .NET Apps with ArcGIS Online and Runtime - Thad Tilton & Antti Kajanus
Just caught the end of this talk but was able to see Thad author a web map of point locations of hotel customers and use ArcGIS Online’s premium GeoEnrichment services to add median income as an attribute. Antti took over the end the demo to quickly write a simple JS application to display this data. This was short and sweet for me but good to see Thad on stage (a former GISi employee ☺ ). Also it was the third time at the dev summit where I saw a demonstration of how the JS API can take a web map ID parameter to create the entire map – so much easier than loading each layer individually!
HTML5 Geolocation: Advanced Topics - Andy Gup
Ever wonder how a mobile mapping application such as Yelp or Google Maps can figure out where you are? Here were the guts of that logic. It’s cool but under the covers it’s not pretty. As a developer, you have to take into account all the variables to come up with the right math to accurately represent a location that will work for your application. HTML 5 will ask the user for access to their location will deliver that information to the app through a geolocation object. The developer must work with the options such as timeout, max-age, and enabling high accuracy and realize that there is no guarantee that the location will respond or will, in fact, be accurate or precise.
OK definitely the weirdest presentation I’ll ever see at a Dev Summit – lots of fun. Patrick started out with lots of examples of how quickly you can get unexpected results without realizing the impact of global scope. He then followed up with a demonstration of how he could write any logic he wanted with only six characters – something like ‘(/+ and !’. There’s no way I could explain it in this short blog but suffice it to say that it was conceived by a hacker to write a bunch of gobbly gook that would slip by any network sniffers with these characters to dynamically create strings and numbers that could be turned into actual running code. Pretty cool – pretty weird – fodder for happy hour.
Declarative Mapping Applications with AngularJS - Tom Wayson & David Bouwman
More slides of Darth Vader and Stormtroopers – sorry inside joke – just enjoy Dave’s slides. This time Dave and Tom talked about options to incorporate the Angular framework into mapping applications that also use the Esri JS API and Dojo. Dave and Tom used Angular to create the ArcGIS Open Data Site and so they have some experience here but they were quick to point out that this site is not a typical mapping application, it’s very much a ‘forms over data’ site which is where they feel Angular shines the most in terms of value. They point out that for a typical mapping application (map in the middle, header, identify, results pane etc.) that it’s probably best to NOT put the Esri map in a directive and instead we should consider building custom directives that are specific to the map’s needs. Perhaps a tough concept to grasp unless you’ve actually been down that bad road (and several folk in the audience had) but the cons of such an approach lead the ignorant developer to create gobs of HTML tags in their view. To demonstrate their approach they created a sample app on GitHub that uses a custom directive for their map, which demonstrates a controller that has all its Esri map logic in one module. Curious? Check it out for yourself at https://github.com/tomwayson/angular-parcel-map. If you are more than curious and have your own opinion they ask that you please share your ideas with them on GitHub as they are considering ways that Esri can support Angular development with the JS API.
The Smart Mapping Initiative for ArcGIS Online - Jeremy Bartley & Mark Harrower
I only caught the tail end of this one – another packed room – however Dan Levine and a few other GISi‘ers were very excited by these new smart mapping features in the ArcGIS Online Web Map author which were unveiled for us at the Dev Summit. Check out their blogs for more info. I paraphrase Dan “This will vastly improve the visualization of the GIS data we see on the web.”
With day 1 under our belts, moving into day 2 there were hopes of cool technical sessions. The day started with a keynote address about the challenges faced when developing large-scale mobile applications.
From there we went right into technical sessions. First up for me was “Advanced tips & tricks for building powerful iOS/Mac apps.” The presenters gave a list of 20 tips. Here are my top five:
1.) Graphics layers render in static mode by default. This builds on the session from yesterday, “ArcGIS Runtime SDKs: Core Display Architecture Performance Tips and Tricks, “ where they described how maps are rendered on the core runtime. Static mode has the same performance no matter how many graphics are on the graphics layer. This makes it ideal for layers that will have thousands of graphics. Conversely, Dynamic mode maintains display textures for each graphic in the GPU, which renders incredibly fast. This works best with smaller graphic sets because having too many textures in the GPU can degrade overall render performance. Set the render mode on a graphics layer to force it to use Dynamic mode.
2.) Use the AGSRequestOperation for downloading files from the server. The class has status updates built-in, which make it really easy to provide feedback to the user.
3.) When performing downloads using any of the Runtime SDK classes like AGSRequestOperation or AGSExportTileCacheTask they will continue to download results when the app is put into the background. If either the operating system or the user kills the app the task can be resumed when the app is started back up.
4.) A notification can be provided to the user if a download task finishes downloading while the app is in the background. Use the [[UIApplication sharedApplication] scheduleLocalNotification] method.
5.) When symbolizing points using a picture the picture is always centered on the point by default. An offset can be applied to shift the picture to a better position relative to the point using the offset property. In addition, the leader for callouts can also be shifted using the leader Point property.
Today kicked off with the Keynote Session (after donuts and coffee, of course). This year’s speaker was John Tomizuka, the Co-Founder and CTO of Taqtile. This is the third Dev Summit I’ve been to, and while I’ve enjoyed the other keynotes, I think John’s talk may have been the best for one reason: honesty. Not that the speakers of years past have been dishonest, but John talked a lot about the evolution of decisions that have been made at Taqtile, and those decisions have at times been driven by problems they encountered with their existing solutions. The best learning experiences often come from failures rather than successes, and so John’s talk about some of the failures that Taqtile experienced, and how they corrected them, made what he had to say especially interesting and informative.
Spoiler alert: the downside to attending a highly-anticipated session right out of the gate is that the rest of the sessions after that tend to pale in comparison.
Thankfully the session after that, “ArcGIS Web AppBuilder: Customizing and Extending”, helped to turn the day around a bit. The two presenters here did a very respectable job of presenting the Developer Edition of the Web AppBuilder in a logical and informative manner. While I had seen some of the information before in an internal tech exchange, there have been some updates since then (it was just in beta at the time), and it was good to hear it explained straight from the people who designed it. For anyone that is familiar with how Dojo widgets are built, extending the Web AppBuilder with custom widgets should be a familiar task, as the app builder’s widgets inherit from Dojo’s. The Web AppBuilder itself is divided into several decoupled components, which will make it easy for developers to extend any one of them without breaking the others. They also started to dive into the theming process, but quickly ran out of time and ended up rushing through that section. So unfortunately I didn’t get to learn much about how to implement custom themes, but overall it was still an informative presentation and I’ll be looking forward to putting some of it into practice whenever I have an opportunity to leverage the Web AppBuilder for a future project.
The day started with an interesting Keynote Session with John Tomizuka of Taqtile. He discussed the growing pains of developing applications and shared some of the technology stacks that they are using to solve their challenges. One thing that always seems to be stressed in lessons learned with regard to application development is the need to take the time to go through a comprehensive design phase.
I attended the User Experience and Interface Design for Web Apps presented by Mike Gaigg and Allan Laframboise. This was a very relevant session to go to after the Keynote. They provided great examples of how to use existing UI design patterns when building applications and Mike has put together a nice site with best practices for map applications.
One of the best things about the Dev Summit is being able to sit and talk with the developers that are making Esri’s applications. I was able to work with Morakot Pilouk Wednesday afternoon and really dive into GeoEvent Server. I learned a quick way to use GeoEvent Simulator to test GeoEvent Services.
In this example, I will show how to test a cache-aware processor in a GeoEvent Service.
First I need to add an input for the GeoEvent Simulator.
I will need to use TCP Socket for text.
Then I need to add an output.
For this, I want the GeoEvent Service to write to a CSV file.
Next I need to create a CSV file that the Simulator will read to mimic changes in my service. The first field sets the name of the GeoEvent definition and the remaining will be present in the service.
Each row in the file will simulate a change occurring in Field1.
Now I can add a GeoEvent Service and plug my input and output together with a processor.
The processor I am using adds a field to the service and calculates the cached value for Field1. In other words, it gets the previous value for a field.
Now I will set TRACK_ID for the GeoEvent Definitions of my processor and input. In this case, I am setting it to Field2. This needs to be a unique id for the data.
Next I need to load the CSV file into Simulator
Then I can start the simulator
and start stepping through the records.
Now if I check my output CSV I can validate that my service is calculating the cache aware field properly.
The Keynote by John Tomizuka of Taqtile focused on saving us from pain. He covered many of his lessons learned over the years from several of the different mobile applications his company has developed. I liked his use of the terms "narcissism" and "empathy" when considering traits of apps. A narcissistic application was not built thinking about the user perspective and can sometimes have useless cool features; you build the empathetic application by putting yourself in the users' shoes.
Day 2 started with a great keynote from John Tomizuka, co-founder and Chief Technology Officer of Taqtile. He went through lessons learned in building innovative and scalable mobile apps and provided a great list of tools for design, development and testing that I will be checking out when I get back in the office. A big focus for Taqtile is UI/UX spending as much as 1.5 months on wireframes. Making that initial investment in user experience and gaining buy-in from the users throughout the development process has made their application adoption rate significantly better. Esri is also focused on UI/UX with a design lab and UX sessions throughout the week. This was great to hear and further validated our (GISi’s) investment in an application designer and increased focus on user experience.
My focus for the day was scalability, big data management, and statistical modeling. Esri is making huge strides in 2015 to optimize the ArcGIS Platform for big data management and analysis. Highlights include R Integration, the Esri Smart Mapping Initiative, ArcGIS Pro geoprocessing capabilities, Hadoop Integration, and client-side processing. My favorite sessions on day 2 were.
Integrating Open Source Statistical Packages with ArcGIS
The R integration includes:
- In Proc Processing
- Native GIS data access
- Honors selection sets and projections
- Access to vector data
- Charts and graphs
- Same GUI interface as ArcPy
The Smart Mapping Initiative for ArcGIS Online