By James Gammon , Bipin Regmi, Jordan Watkins & Carson Whitt
INTRODUCTION TO CROWDSOURCING
The practice of crowdsourcing can prove to be a powerful agent in the development of effective and sustainable software. Crowdsourcing enlists the services of a large group of people to achieve goals that would otherwise be impossible for computers. It utilizes the human experience and our innate understanding of how the world works to supplement computer programming.
There are several companies and websites that harness the power of crowdsourcing. One such website is Amazon’s Mechanical Turk (MTurk) which hires remote “crowdworkers” that are contracted to complete tasks known as HITs (Human Intelligence Tasks). These HITs offer a monetary incentive for MTurk employees but often pay far less than the average hourly minimum wage rate in the United States. Other companies that utilize crowdsourcing offer alternative incentives including entertainment, research contributions, and the satisfaction of contributing to a humanitarian cause. We plan to harness the power of crowdsourcing and develop our application design to utilize MTurk’s pre existing website.
The integration of crowdsourcing in our application will allow us to get a wide breadth of data from a large population while keeping the amount of work for each individual contributor to a minimum. We will outline our design process and discuss crowdsourcing more completely below.
TEACH MY CAR
Our application, Teach My Car, operates as a data reception tool for companies developing autonomous vehicles. The application utilizes Amazon’s MechanicalTurk (MTurk) to employ workers to do simple identification tasks relating to the rules of the road and how they would operate in the case of an uncertain traffic condition. The application is laid out in a simple quiz format, showing the user one question at a time that can be answered in a multiple-choice fashion.
Answers from these questions will be collected and the data generated will be sent to companies that are involved in the development of autonomous vehicles. This data will add a human understanding of how the road works and provide supplementary input to be used in the training of autonomous vehicles. We want to positively impact the decision-making skills of these vehicles while utilizing the experience of real people through crowdsourcing with MTurk.
The importance of our human influence approach lies in situations that a data trained autonomous vehicle may not be programmed to handle properly. In many situations, humans must briefly deviate from standardized rules of the road in order to either drive safely or efficiently. Take an ambiguous situation in which a driver is stuck behind a garbage truck, for example. The garbage truck is taking up half of the two-lane road which is separated from oncoming traffic by a double yellow line. The truck is stopping at each sequential garbage can, resulting in an abnormally long trip for the driver if they do not make a decision that will violate the standard rules of the road. The autonomous vehicle will drive behind this truck for as long as the truck is ahead of it or the road conditions change. In a situation like this, a human would automatically check oncoming traffic and cross the double yellow to get around the truck. With the influence of human experience implemented alongside autonomous vehicle driving algorithms, both safety and efficiency could be improved and situations that would normally be ambiguous for self-driving cars could be handled appropriately.
Our report below outlines a team-built storyboard design process that utilizes the power of crowdsourcing to create an environment that captures the benefits and potential applications of human-computer interaction in the development of successful software. We decided on an idea that both excited us and challenged our understanding of the field of machine learning. We’ll discuss the multi-step process of brainstorming, developing, and evaluating this design below.
Step 1: Ideate & Critique
As seen in the figure above, we started our development process with a period of individual ideation followed by a period of critique. Each group member developed a list of potential ideas. We conducted this step with a stream of consciousness approach to get as many ideas on our papers as possible. Once these lists had been created, we went on to critique them and ultimately remove ideas that were outside of our scope of abilities or did not fit the requirements of what we were trying to fulfill. Below we have provided a couple of the lists that our team members developed so that you can have a good idea of the range of options we explored during this phase of development.
Step 2 : Idea Decision
Once we refined our individual lists of ideas, we came together to discuss possibilities and determine which ideas would best suit our vision for an application that spotlights crowdsourcing. Those ideas were condensed into a single list, with favored ideas appearing first.
Due to a common interest in autonomous vehicles shared by all members of the group, we were able to make the decision to pursue our top idea of humanizing autonomous vehicles. This strong interest paired with the opportunity to address a problem that could impact a soon to be ubiquitous technology maximized our motivation to pursue this idea. We began formulating ideas and decided to take the preconceived notion of an autonomous vehicle and add a flavor of humanity to it through the application of crowdsourcing and subsequently, human experience.
STEP 3 : Storyboard Development
It was at this point in our development process that we created our first storyboard, driven (pun intended) by the idea that data received from a crowdsourcing platform could be used to create an autonomous vehicle with supplemental human influence implemented. Check out our elementary storyboard below —
Our storyboard above is certainly rough, but it captures an integral part of our design process. This first draft was primarily developed to create a consensus between our group members to ensure we were all approaching the development process with the same vision in mind. The prototype focuses on the user interface that users would be interacting with. It shows a start screen that launches the user into one of three possible options; a quiz, a game, or a camera-based “training” mode. We implemented an additional “hot-fix” screen in the bottom right frame of the illustration that allows users to view updates and application news. This early prototype does little to capture the applications dependence on Amazon’s MTurk and fails to outline its dependence on crowdsourcing. We noted these missteps and went on to refine our design to more adequately represent our vision for this product.
Step 4 : Feedback Period
After creating this early draft of our prototype and receiving some much-needed feedback on our design, we were ready to design a storyboard that more effectively purveyed our goals for this application. We were advised to promote more of a story rather than just showing off a user interface as we had done in our original draft. We also felt the need to include some mention of MTurk and provide viewers with an illustration that made the frames easy to parse through. Our original prototype was difficult to follow due to the way that our frames were laid out, so we made an effort to make the transitions as intuitive as possible for the viewer. We utilized our feedback, made the appropriate changes, and created the following storyboard.
As you can see, a lot of changes were made to make this storyboard a more effective tool for describing our design decisions. We approached this version with the intention of fleshing out a story that outlines the benefits of our application for our two participating parties; those that will interact with our product and those that will utilize the data produced from it. The first frame addresses the dangers of autonomous vehicles and shows our protagonist’s interest in contributing to the safety of them. The second frame depicts our protagonist discovering a Teach My Car HIT offered on the MTurk website. They decide to click on it, at which point they are met with a start screen that sends them into the refined version of our user interface. Instead of offering multiple options for data collection, we decided to utilize the form of collection that made the most sense for our purpose; simple quizzes. Once the user presses start in the menu screen, they are sent directly to a quiz screen which can be viewed in the fourth frame. At this point our users will answer a series of questions that last, on average, just below five minutes.
Upon completion, they will be notified of the amount of money they have earned and be given the option of completing another quiz. There are more features outlining specific numbers, including the total amount of contributions, that can be viewed on this screen (frame 5). The storyboard also shows a screening process that occurs behind the scenes in the last frame of the illustration. This scene depicts a Teach My Car employee that is monitoring the legitimacy of the answers provided by our “crowdworkers” to ensure that the data we provide for autonomous vehicle developers is honest. We will include a “gag question” in each quiz that, if answered incorrectly, will automatically disqualify the quiz and result in the “crowdworkers” payment being withheld.
At this point in our process, we felt confident that our storyboard reflected our goals for this project well and decided to move into the final phase of development; the presentation period.
Step 5: Presentation Period
At this point in our process, we were confident enough in our design and the way that we showcased it to bring it to some of our classmates, professors, and family members. We presented our process and consequent design and prompted them to provide us with feedback. We asked them if they could see themselves taking a moment to contribute to our application and many were interested by the ease of use and monetary incentives offered. There were concerns regarding our ability to connect with companies that would be looking for the data that Teach My Car would provide. According to an article from CB Insights Research, there are more than forty corporations worldwide that are in the race to develop an efficient and safe autonomous vehicle (CB Insights). We believe that the data sets that we will produce can contribute to these competitors’ missions and bring out the best in road safe autonomous vehicles. As we continue to present our work to new audiences, we expect to continue receiving feedback and plan to refine our design accordingly.
Secondary Features
As we designed the main features and functionality of Teach My Car, we realized that we wanted “crowdworkers’” experiences to be equivalent to that of employees working in other minimum wage jobs. After watching a short video called “Turking For a Living”, we realized that workers contracted by MTurk are paid far below the national minimum wage average. We decided that we did not want to be a company that takes advantage of loose contractual working legislation. Instead we made the decision to provide our employees with the compensation they deserve for the work that they complete for Teach My Car. In order to achieve this we have adopted a function that will adjust the payout of our HITs on MTurk based on the current minimum wages in the United States and the average time of completion for a HIT. The function operates as follows:
Minimum payout per HIT = pph(minWage{$/hr}, avgTimePerHit{hr}) = minWage * avgTimePerHit)
- pph = the function name that calculates Payout Per HIT to crowdsource workers
- pph() takes two arguments. The first being the minimum wage currently in the US (In dollars per hour. The second is the average time it takes for “crowdworkers” to complete a single HIT based on data processed by MTurk (In hours).
- To calculate the payout per HIT we simply take the time in hours to complete a HIT multiplied by the current hourly minimum wage.
- This function ensures that “crowdworkers” completing Teach My Car HITs receive a fair payment for their contribution. In addition to providing fair wages, this will incentivize “crowdworkers” to continue accepting Teach My Car HITs and result in improved data collection efficiency.
Implications for Design
Upon completing the development process for this application, we realized the importance of creating a mutually beneficial experience for all parties involved in a crowdsourcing relationship. With lack of effective incentives, “crowdworkers” often find themselves unmotivated to complete tasks that they view as a waste of time and effort. If we could give any developers advice when considering the application of crowdsourcing, in our limited experience, we would suggest ensuring that incentives are made worthwhile for those involved in completing tasks. This applies to the parties benefitting from the tasks being completed by “crowdworkers” as well, although to a lesser degree. Tasks that do not provide a worthwhile incentive for the amount of work required are often denied by “crowdworkers”.
Some tasks do not fit within the scope of crowdsourcing, most notably those that rely on the opinions of the “crowdworkers” completing the tasks. We found an article that describes some crowdsourcing implementation failures, all of which ask the public to deliver an opinion on something. We have found that it is best to offer “crowdworkers” tasks that are not open to interpretation and instead can be completed alongside a set of rules. Because crowdsourcing utilizes the skills of such a large amount of people, a lot of work can be done quickly, leading to the efficient completion of a large project that would normally take hundreds of hours of work for a smaller group of individuals. Because of this, projects that require a multitude of small contributions to complete the desired goal can be effective candidates for crowdsourcing.
Generally, the most common applications of this powerful tool are used to complete tasks that computers are unable to do. MTurks HITs are appropriately named because they utilize human intelligence to complete something that is not within the scope of a computer’s abilities. With our refined understanding of crowdsourcing in tow, we were able to design a storyboard design that effectively utilizes its principle function. We provide autonomous vehicle developers with human intelligence data that would normally be impossible for computers to gather themselves.
Evaluation
Our idea works effectively due to a multitude of strengths that help create accessibility and general enjoyment for many audiences, including providing a large incentive for “crowdworkers” via high wages. We plan to offer higher wages, on average, than other HITs offered to “crowdworkers” on MTurk. In addition, our quiz questions only require common knowledge of a widely used skill so it requires minimal effort from our “crowdworkers”. The answers generate concise data to be utilized in the development of autonomous vehicles which ultimately creates a safer road in the fast-approaching world of ubiquitous autonomous vehicles.
Despite the potential effectiveness of our product, there are several weaknesses that we plan to improve upon in the near future. As we strive to create a more employee-centric product, we hope to include employee benefits by working with Amazon to leverage these perks for our crowdsourcing employees. If we are unable to forge an agreement, we plan to develop our own crowdsourcing network that will be able to include benefits for our employees. We also fear that the questions in our quizzes may become stagnant. Therefore, we intend to create a complementary application to Teach My Car that prompts users to input situations that could be utilized in our base application. With that idea comes a variety of new challenges that we will omit from this report for the sake of our readers’ sanity.
As our design process proceeds, we plan to continue increasing the scope of availability to users and expand to countries outside of the United States that utilize personalized settings to appropriately fall in line with other countries driving standards (i.e. driving on the left side of the road). We have a long road (again, pun intended) ahead of us in the journey to bring Teach My Car to a state that we deem acceptable to release to the public. However, we are confident that Teach My Car could positively impact the lives of our employees while creating a safer road for both humans and machines alike.
Conclusion
Teach My Car approaches the world of autonomous vehicles with a focus on how humans react to adverse or unexpected road conditions as compared to how these vehicles would. We intend to provide developers with data that can positively impact these vehicles’ development to increase both their safety and efficiency. We utilize the power of crowdsourcing through Mechanical Turk to collect this data. Our five-step design process for Teach My Car describes our efficient methods for the formulation of ideas, reception of critical feedback, and refinement of those ideas into a polished product. We plan to continue tweaking this design to bring us to an optimal state of data collection and collaboration with autonomous vehicle developers.
We thank you for taking the time to review our process and request that you feel free to provide critique or ideas for the next steps or refinement of the product in the comments below.
Welcome 