Abstract: The article presents how applications programmer can access data provided by sensors on Android smart phones and tablets. It discusses the limitations of the current paradigm, both in terms of the sensor subsystem performance, and in its ability to provide useable information to an applications programmer. The author also outlines the possible system improvements needed to address these deficiencies.
Available Sensors on Android
It is surprising how many physiological senses are represented on a smart phone or tablet. The camera, the most widely-used sensor on a phone, allows a device to “see” the outside world, while the microphone lets the device “hear.” And many devices now have multiple cameras and microphones for possible spatial resolution.
Other means to let a device “see” include light and proximity sensors. The ability to “hear” is further facilitated by any of the four wireless sensors (cell tower, Wi-Fi, Bluetooth, and GPS). Considering these communication devices as sensors naturally leads to methods of
determining context, and providing an overall “sense” of the world around the device.
There are even more senses available on smart phones and tablets:
• The magnetometer allows a sense of direction, nominally pointing towards magnetic north;
• The accelerometer and gyroscope provide a sense of balance;
• The pressure and temperature sensors, as well as the touch screen, build on the sense of touch.
The only senses not represented on a mobile device, at least for now, are the chemical senses of taste and smell.
Effective utilization of these sensors is still in its infancy. Android provides a framework to access all these sensors at the application level. This encourages developers to create new uses for the available hardware. For example, a weather prediction app can detect changes in barometric pressure, or a health monitoring app can measure a person’s pulse using the accelerometer. However, the lack of good high-level API’s across all sensors is still a stumbling block for app developers.
In Android, the cameras and microphones have the most structure in their API’s, with access not only for container protocols such as JPG and MP3, but also for fundamental and interpreted data. Raw camera parameters are available in the android.hardware.Camera package. This provides methods for applications to retrieve the currently detected focal distance getFocusDistances() and scene mode getSceneMode(). As camera hardware develops more algorithms to take better images, the developer can use the information for other purposes, such as determining the depth of field or overall perceived context.
Other algorithms for pictures are provided in the Android OS itself, such as android.hardware.FaceDetector which identifies a face in a bitmap graphic. Although this is a rudimentary detection of two eyes at this time, a framework exists to roll out more sophisticated algorithms.
Raw audio information is available in android.media.AudioRecord. This provides the pulse-coded modulation audio directly from the hardware that can be manipulated or analyzed by the applications developer.
Some algorithms to interpret sound are already provided, such as android.speech.RecognizerIntent, which accesses a cloud-based speech-to-text service for Android devices.
The Wi-Fi sensors are accessible for determining location using the network android.location.LocationProvider. This is a cloud-based database of cell towers and Wi-Fi access points provided by Google.
All the other sensors, beyond sight, sound and location, are collected in android.hardware, and accessible using Android’s SensorManager . These other sensors are primarily used for motion tracking as found in augmented reality apps or games, and will be discussed below. A framework for high-level API’s to interpret this data is done through virtual sensors.
The currently adopted set of virtual sensors in Android were introduced in Gingerbread. For example, an accelerometer measures 3-dimensional acceleration of the device in m/s^2. This acceleration includes user-generated motion (both translation and rotation) and Earth’s gravitational field. Algorithms and at least one independent sensor are needed to properly disentangle these. Once done, these can be represented to a user as TYPE_GRAVITY, TYPE_LINEAR_ACCELERATION, and TYPE_ROTATION virtual sensors.
Issues with Using Sensors in Android
To determine the sensors available on a particular device, an instance of the SensorManager can be created and used in an Android app as follows:
SensorManager myManager = (SensorManager)getSystemService(SENSOR_SERVICE);
List<Sensor> sensorList = myManager.getSensorList(Sensor.TYPE_ALL);
StringBuilder sensorString = new StringBuilder("Sensors:\n");
for(int i=0; i<sensorList.size(); i++) {
sensorString.append(sensorList.get(i).getName()).append(", \n");
}
Running this on multiple devices shows some important differences.
For example, the HTC EVO 4G has:
BMA150 3-axis Accelerometer AK8973 3-axis Magnetic field sensor AK8973 Orientation sensor CM3602 Proximity sensor CM3602 Light sensor
whereas the Samsung Nexus-S has:
KR3DM 3-axis Accelerometer AK8973 3-axis Magnetic field sensor AK8973 Orientation sensor GP2A Light sensor GP2A Proximity sensor K3G Gyroscope sensor Gravity Sensor Linear Acceleration Sensor Rotation Vector Sensor
Comparing the sensors on these two phones demonstrates the sensor fragmentation now found in Android:
1) Non-standard sensor availability: The Nexus-S has a gyroscope (from ST Micro), but the EVO does not. In fact, most Android devices do not have a gyroscope. There is no standard availability of sensors across devices.
2) Non-standard sensor capability: The BMA150 is a Bosch Sensortec 10-bit accelerometer, and the KR3DM is a ST Micro 12-bit accelerometer (using a special part number). In fact, there is no standard capability requirement for sensors across devices to ensure consistent
resolution, noise floor, or update rate.
3) Sensors not fully specified: The AK8973 is an AKM magnetometer, which is only 8-bits. Analyzing this data stream shows it is low-pass filtered. This fact is not published on the phone or even the sensor datasheet. Many sensors have characteristics not specified such
as bias changes, non-uniform gain, and skew (coupling between measurement axes). Algorithms that use sensors without knowing these extra characteristics may produce incorrect information.
4) Broken virtual sensors: The AKM sensor driver abstracts out an orientation virtual sensor which is derived from the combination of two sensors: the accelerometer and magnetometer. However, support for this virtual sensor was dropped early on, so the TYPE_ORIENTATION
sensor is deprecated and the method SensorManager.getOrientation() should be used instead. Furthermore, the new virtual sensors introduced in Android 2.3 (Gingerbread) are not supported on all devices.
The sensor differences between just these two phones is substantial. So when a developer is faced with writing apps utilizing sensors across as many devices as possible, it is a daunting task.
Furthermore, the Android platform is not optimized for real-time sensor data acquisition.
Sensors are by nature high-interrupt and low-latency. Every time a new measurement is available the sensor interrupts the core with the information. Any delay in servicing and using the data can disrupt motion tracking or game response.
Since Android is not a real-time operating system (RTOS), some samples can be delayed, leading to incorrect timestamps, or even dropped when the core is busy. For example, when the Dalvik garbage collector kicks in, sensor data can be lost for as long as 200ms.
Methods to Improve Android
As more sophisticated uses of sensor data are implemented, the above issues become more apparent. No app developer can characterize and support all different sensor types in order to provide a consistent user experience across all platforms. Standardization of a sensor suite to unify the capability and availability of sensors is ideal, but would take time to be realized.
Simple hardware to assist real-time data acquisition: Rearchitecting the platform to service sensor interrupts in a timely manner is another method to improve the experience. To support the broadest application requirements, smart phone system architecture can
be supplemented with hardware to assist in real-time sensor sampling. Hardware assistance can be a sensor hub, with dedicated computational resources for the sensors, or just a smart DMA engine to perform the real-time data collection. This is analogous to what hardware accelerators and GPUs do for gaming and video applications. Another method is to distribute critical portions of the high-interrupt computing into the microprocessor of the sensors themselves.
Improved algorithms: Good, efficient algorithms are necessary to enable these architecture changes. Algorithms can also ensure seamless repeatable performance across different sensor manufacturers and overcome variations in sensor performance by combining
sensor information.
Even with perfectly designed sensors and core logic hardware, performance of a sensor platform ultimately rests with the intelligent algorithms and heuristics. As an analogy, consider the maturity of hardware acceleration to enable camera display and video processing. Yet app developers still need algorithms for interpreted features such as facial recognition or landmark tagging. In the case of the motion-detection sensors in Android, such interpretation takes the form of virtual sensors.
More virtual sensors: Android provides a start at standardizing the virtual sensor framework so that any applications developer can utilize them. The provided algorithms are of minimal performance, but more sophisticated algorithms can fit in the framework such as step detection, in-elevator detection, and augmented reality icon movement handler. Furthermore, variation in performance from platform to platform can be minimized with good algorithms as well.
Conclusion
Mobile devices have a surprisingly complete set of sensors. Android provides a nice framework for developers to access raw and interpreted sensor data. However, a solution to sensor fragmentation and improved virtual sensors are still needed to enable developers to leverage this data.
Photo Credit: Nasa (Chemical Cellphone Sensor)
Related posts:
- Pyxis Mobile first platform to offer native Android support
- WorkLight Becomes First Mobile Application Platform to Offer Multi-Platform IDE for Combined Native and Web Coding in Single Application
- Mobile commerce platform planned by PayPal and Appcelerator
- Appcelerator and FortiusOne Partner to Unveil First Mobile Location Analytics Platform
- Kanzi Enables Rapid Creation of 3D UI’s for Android Devices
Recent Comments