Balancing CSCW with Individual Work Through IM

Written Dec 2012

Introduction

The knowledge worker today is tasked with both autonomous and collaborative work. Their environment includes many forms of Computer Mediated Communication (CMC) that help support their needs in Computer Supported Collaborative Work (CSCW), the most prevalent being email, phone, and Instant Message systems (IM).

Email best supports asynchronous communication through messages sent between individuals or groups. Depending upon the system, the recipients may receive a notification of an incoming message and can reply at anytime. Users typically have several concurrent email conversations that can last for an extended time. Though widely used, email does not signal awareness and availability, messages are delayed, and the expectations for responsiveness are much slower than other channels like IM and phone [5].

Phone supports directed synchronous communication that allows parties to collaborate in real time, but in most cases requires the full attention of the participants. The response time for answering a phone call is extremely limited; if the recipient does not answer before the sender hangs up, the sender can either try again later or leave a voicemail, thus putting pressure on the recipient to answer.

Though not a form of CMC, Face to Face (FtF) is another synchronous channel which has similar characteristics to phone in that it requires both parties to devote their primary attention to the conversation. It too has limited response time expectations because FtF interruptions are nearly impossible to ignore.

IM best supports synchronous communication through messages sent between individuals or groups. The recipients receive an audible or visual notification at the time a new message arrives, which often interrupts work but does not require an immediate response, though it is hard to ignore [6]. IM involves variable wait periods for responses, so users can perform other tasks during the course of a conversation, including having other IM conversations. In addition, the typical IM system includes information on the user’s status, which can be used to determine if they are available to converse.

Theoretical Assumptions

Handel et al. reported that synchronous messages are typically used in three scenarios; opportunistic interactions, broadcasting of information or questions, and as a way to negotiate availability [5]. Each of these scenarios are helpful in collaboration, which often makes synchronous messages the most efficient way to collaborate. IM supports nearly real time interaction, notifications of new messages, and availability information. The real time interaction provides similar collaboration characteristics as phone or FtF, the notifications work as an effective signaling mechanism and the availability information allows senders to make informed decisions on how interruptible a recipient is. In addition, IM provides the ability to multitask, even in times of other CMC use and allows multiple collaborations to happen at the same time, potentially greatly enhancing the work done within the same timeframe [7].

A concern with any synchronous communication channel is the burden of interruptions that they introduce. While IM may seem burdensome, Ou et al. found that IM accounted for only 5% of workplace interruptions and that FtF, meetings, email, phone etc. were more prevalent interrupters. This suggests that interruptions are part of the workplace and will happen with or without IM use [7]. Also, Garrett et al. found that those that use IM at work reported being interrupted less frequently than those that did not use IM [3].

Another concern is that there are too many tools in the workplace, but Redmiles et al. found that users were able to use many systems in concert and that users transitioned through media as their conversations evolved. Their report emphasized that people can use off the shelf software and that problems may be less prevalent than is typically reported due to underreporting the positive use and adoption [9]. In addition, Ou et al. found that having integrated use of different CMC technologies had a substantial effect on communication performance over a single tool [7].

Assuming that IM has an appropriate use in the workplace, how do workers use IM to balance their collaborative work with the individual work they are expected to perform? How have users adapted to the tools and how can the tools evolve to better support the users’ environment and actions?

Methodology

To research the topic, I interviewed 10 people who use IM for work at least once a week. To screen participants, I created a survey that was completed by 28 people. The interviews were conducted over a three week period and lasted between 20 and 45 minutes. The participants were from 6 companies ranging in size from 6 people to many thousand. The age range was 25-35 and there were 6 men and 4 women. Half of the participants filled a managerial role while the other half were primary contributors.

Results

Through my interviews I found that IM systems were used for 4 main purposes; for quick questions, to initiate other forms of communication, as an escalation from other CMC, and to determine and manage availability through IM status.

The subjects stated that they often used IM to ask and answer questions that were simple and fast to answer. They preferred IM for this because unlike FtF it allowed them to stay at their desk, they felt that it was less disruptive than a phone call, and unlike email people responded quickly. One participant said they would start with IM when they did not think the conversation would take very long and when the conversation could be completed without distracting the other person too much, particularly when they knew the other person was generally busy.

My subjects found that IM was a good way to negotiate availability and used IM to initiate FtF or phone conversations in cases where they felt a dedicated, focused synchronous conversation was more productive than one over IM. Likewise, the subjects preferred to be contacted via IM before a FtF conversation because it gave them time to prepare for the interruption and complete the task they were working on. Subjects voiced frustration at people interrupting them at their desk without warning.

Subjects used IM when other forms of communication were not getting the desired response or resolution. For instance if an email had not been replied to in the desired timeframe, they would reach out via IM to either request a reply or to re-ask the question to get an immediate response. Multiple subjects reported that they used IM to clear up misunderstandings from emails.

The subjects’ IM systems all included information on a user’s status. Statuses were used for three main purposes; to see if someone was likely to respond to IM, to determine work state, and to see if another user was at their desk and available for another type of communication.

Subjects stated that the status most signified the responsiveness of a potential recipient; it set the sender’s expectations for how quickly the recipient may respond. Several subjects said that if someone was marked as “busy” they had little expectation that the recipient would get back to them quickly.

When used to determine work state, subjects looked at someone’s status to determine how long they had been away or offline to then determine if they were working, had been working that day, or likely to be back online later. The subjects then used this information to determine how best to get the information they were seeking. Most subjects felt that it was particularly important to be online when working remotely to make it clear that they were working, as well as provide an easy channel for communication. When used to see if someone was at their desk, subjects would see if someone was marked as “available”, infer they were sitting at their desk and would initiate a phone call or walk to their desk. Lync, a popular IM program with my subjects, will automatically set a user’s status to “in a meeting” if their calendar has a meeting scheduled. In these environments, subjects felt that an “available” status typically meant the other person was at their desk.

Subjects were aware that their status conveyed their availability and felt that manually setting the status to something other than “available” led to fewer conversations and helped filter out the less urgent interruptions. Fifteen of the eighteen people that responded to a survey question on interruptions said that they would consider the recipient’s status before sending an IM. When they encountered “Do Not Disturb” and “busy” statuses, some subjects indicated it led them to do more research, consider contacting another person, or changed the tone of their outreach to be more tentative or apologetic.

Nearly all of the subjects said that they read their new IMs as soon as they saw them, though one subject reported that they did not immediately read messages when they were already in a conversation with the sender and the context was not urgent. Subjects felt an obligation to help others, but felt that it was acceptable to sometimes delay responses. All participants felt they had at least some obligation to be available to others via IM, and most felt obliged to respond within a few minutes and likewise stated that they would expect a response from others in 5-10 minutes if they appeared available. One subject stated that their obligation to respond to an IM got worse the longer they waited to respond, but at some point the urge receded and they then felt that an answer was probably no longer needed. When a recipient was the best source for an answer, they said that they were more likely to respond quickly.

Subjects stated that they preferred IM for CSCW over other channels because they felt it was less of an interruption, that it took you out of the context less than phone or email and it could be ignored or delayed if the recipient was busy. Subjects found that they could ignore messages until they could move to a time of lighter concentration. Furthermore, many felt that it was more conducive to multitasking than other channels, and that they were able to hold multiple conversations at once. In an IM conversation, the receiver can often assess the priority of the communication before engaging the sender. Several subjects reported that they would be more likely to answer an IM than a phone call because they could prepare for the interaction better, and IM was an effective way to ease into a conversation

They also found that it was difficult to predict how long a conversation was going to take prior to picking up the phone, and that once you answered, you were committed to a conversation. One participant preferred IM to phone because they were able to re-read the history and regain context for the conversation, also they found that they were able to follow the conversation more effectively.

A few of my subjects preferred IM over Face to Face (FtF) communication. When one subject was angry with coworkers, they would use IM because it gave them time to more thoughtfully respond and they were able to be less condescending. Another stated that they do not like to go to others’ desks because they are not sure the other person will be available. Likewise, they disliked it when people came to their desk because it was hard to say “no” to someone’s face, and people often did not take “no” for an answer and would ask their question anyway. Also they found that through IM they could stay on topic and ignore the questions that did not pertain to the immediate need, which is not always easy to do in FtF.

Issues

I found several areas that subjects found unfavorable in their use of IM including; people being unresponsive yet online, people offline or not using IM, receiving too many interruptions, and being unable to assess priority before responding to a message.

Subjects stated that they expected a 5-10 minute response time to messages when a user was marked as “available”. There were a few reasons why subjects themselves did not respond within that timeframe, the most prevalent that the subject was working on something of higher priority and did not have time to converse with someone on another subject. Also common was the case where a sender frequently asked questions that were easily answered through other means (documentation, online, general knowledge) or in cases where the subject had answered the same question for the same sender before. Subjects also delayed responses when they felt that the sender had not done enough on their own, or that the sender was taking advantage of the subject’s time and attention. Another case resulting in a slow response was where the recipient already had too many IM conversations active and was unable to focus on another one.

In dealing with slow responses, subjects and their coworkers had three main courses of action; wait for a response, escalate to another channel, or escalate to someone else.

In the case of waiting, subjects stated that if their need was not urgent, they would move on to another task and wait for a response When escalating to another channel, the subjects stated they would chose the next channel based upon the urgency of the need as well as the perceived response time for the other channel. For example if they knew that the recipient was generally very responsive to email, they may escalate to email. There were cases where senders would escalate to FtF communication, which subjects found particularly frustrating. Subjects were most concerned when senders escalated to someone else, because by not responding, they had pushed more work onto someone else, and two people were now disrupted. There were a couple examples where subjects responded quickly to explicitly avoid this case, but it was typically only for certain senders who notoriously escalated quickly.

I found there were two main reasons people logged out of their chat programs. They did not want to be interrupted because they were trying to focus, or they did not want to be interrupted because others could see their screen.

While some stated that they never logged off, others logged off if their current work was more important than the average interruption that they received, and had no qualms about doing so since their job was to produce, and interruptions sometimes hindered that ability. Those that logged off did not feel that they were cut off from all communication and were confident that there were other means through which they could be contacted, or that there were others who could fill in for them in their online absence. One subject said that when they knew they were going be offline, they would send an email to their team giving alternate contact information. In another case, a subject would log off of the company IM system, but still be available through another IM system that a few coworkers also used, thus being available to a select audience. Those that logged off when others were looking at their screen typically did so while projecting in a meeting and wanted to protect against the case where an incoming message would be displayed on the screen and embarrass the recipient and possibly the sender.

Several people responded that there were others who they were never able to contact via IM. When people do not use IM in an environment where it is the norm, it can have two impacts; they cannot be reached, resulting in some other form of communication and they cannot IM others, so they must use a more disruptive approach when collaborating

Like in the cases of unresponsiveness, I found that subjects would use other channels or other people when their primary recipient was offline. If the subject knew that the person was truly available, they would reach out through another channel to ask them to sign on to IM.

Subjects found that when marked as “busy”, users reached out to them tentatively, with little context through which the receiver could determine a message priority. This vagueness led to a couple possible behaviors, first, the recipient just ignored the IM based on previous conversations, second, the recipient responded, but was then engaged in a conversation that could prove to be lower priority than the work that was interrupted.

Subjects reported that there were times when they were interrupted too much and it affected the other work they were expected to do. Subjects typically managed their interruptions more closely near their deadlines by logging out, being online but unresponsive, or telling senders they could not collaborate at the time.

The managers reported the most interruptions, while the primary contributors talked the most about reaching out to others. When subjects were the sole provider of certain information or when the interruptions were appropriate, the subjects were more receptive to being interrupted, as opposed to when getting asked questions that were unnecessary or when they felt that the sender was relying on the recipient to do the work for them.

One subject was a member of a team that only used IM for communication within the team, but the team played a support role to a much larger portion of the company. The subject voiced strong concerns for expanding IM to the greater group and therefore being exposed to the potential for too many messages and thus making IM unusable for the group communication.

Discussion

Subjects indicated that their use of IM was similar to that reported by Ou et al., in that it supported their other forms of communication and allowed them to multitask [7]. Unsurprisingly, I found that each person had different preferences and that their environment shaped their IM use. Some workplaces heavily used status to indicate current work state, while others just relied upon the automatic statuses provided through the system.

In addition to having to balance their personal deliverables with their CSCW, users also had to meet their workplace expectations for being available. Nearly every subject mentioned that there was at least some expectation that they were available via IM, though I heard of no policy that was in place to ensure it. This shows that the communities of users have adopted what works for them to meet their workplace needs. The tools and features did not appear to be used just for the sake of it, for instance I found features like the “urgent” flag in the Lync application that were unused, even though its email counterpart was. This may reflect that users found the similar email urgency flag was abused, so its use was not carried over to IM use.

Different users of IM have different needs and behaviors. I found that the managers were more available for interruption and voiced stronger obligations for availability. Individual contributors had less expectation for availability and were more likely to have large blocks of time where they were “in the zone” and not as open to interruptions. One manager stated that they were in and out of meetings and their work was interrupt driven anyway, so distractions of IM did not bother them. The same subject felt that as a manager, it was their job to be available and that if they were protecting their time from their team or other managers, they were not fulfilling their full role as a manager.

I found it interesting that all of my subjects used status regularly, yet in earlier research, users questioned the value of awareness information [6]. I think this shows how users’ behavior and needs have co-evolved with the tools to better meet the CSCW needs.

Recommendations

When a user is offline, it conveys a few possible scenarios to potential senders; the person is not working, they are presenting in a meeting, or they are busy with other work. If a working user could better reflect they were working and busy, it would help others weigh the priority of interruptions and decide how best to proceed. The “Do Not Disturb” feature of Lync allows a user to reject incoming messages or send the messages directly to email, which provides similar benefit to logging off, in that it eliminates interruptions, but does not have the negative effects that come with being offline. This feature seems to address the needs, yet few of the Lync users stated they used the feature. To help improve awareness and adoption, users could be prompted with a dialog to ask why they are logging out and present the “Do Not Disturb” feature as an alternative. To better support presenters, the software could detect when a projector was being used and move to a less disruptive mode to not display text of an incoming message through a pop up.

Online, unresponsive users are typically working on higher priority work, in too many concurrent conversations, or past conversations have set a precedent for wasted effort by the receiver. To properly reflect that higher priority work is taking precedence, the current “busy” status in Lync and other systems is a reasonable solution. The “busy” status creates a barrier to message and some users said they would think twice about their interruption before sending to someone with this status. When sending a message to a “busy” recipient, senders’ expectations for responsiveness were decreased. To address too many concurrent messages, an IM system could automatically set a “busy” status when a threshold of message windows were open. Another possibility is to infer activity through window switching, mouse movement and keyboard use, and set a busy status appropriately. Tool features are not likely to address the case where senders frequently send requests that the recipient feels are inappropriate. Instead, we should look to the culture and environment, and ensure there are other means through which users can get information without going to others and that there is a culture of self sufficiency.

In the case where subjects were unable to properly assess priority without first responding to the sender, it would be helpful if people started IMs with their urgency, the expected response time, and the expected effort needed by the recipient. Urgency would be in the eye of the beholder, but at least this could serve as a starting point. The expected response time would let the recipient decide how quickly they should respond and the expected effort gives a general sense for how long the interruption will take. Another possibility is to allow the sender to determine how to interrupt the recipient. If the need was urgent, they could create a pop up that is visible until responded to, for less urgent messages they could create a one time notification that would be less disruptive. This could certainly be abused but if in addition the recipient could block certain people from using this feature, it might be helpful. Lync already provides a feature to mark IMs as “high priority”, but none of my subjects indicated they had used this feature.

I found some users did not like to manually set their status because they often forgot to change it back to available. One way to better support the user would be to create a temporary status that had a configurable expiration time.

To address cases where recipients delay a response because they are busy and the sender then goes to someone else, an “answer this later button” could be provided, which would allow the recipient to push a button that sent an auto response to the sender saying that the recipient would answer in a while. The benefit to this is that it clearly states that the sender does not have to go somewhere else and also lets them know it could be little bit of time before an answer is provided. It helps the recipient by letting them quickly respond and not completely lose focus.

I did not find users who felt the statuses were not detailed enough, but Wiese et al. and Ruge et al. argue for more detailed statuses that would allow users to make better informed decisions [11, 10]. The extra detail could indicate if a user was in the office, at their desk, or even at their desk with a visitor. Wiese et al.’s “myUnity” system gathers data from many sources like motion sensors, phone locations, calendar availability, IM status, etc. and aggregates them to create an overall status. To manage their privacy, users can select which data sources can be used. The highest rated features of the myUnity platform were presence state (available, away), location (at home, in the building) and calendar information (in a meeting, on vacation, etc) [11].

While there may be scenarios in which added status information helps senders contact hard to reach people or interrupt at appropriate times, it has another effect on the recipient in that it prevents them from deflecting others through socially accepted “deception”. With many current IM systems, there is ambiguity in a person’s status (even when indicated through the software), which allows the sender to make the assumption that the other person is busy. Several subjects stated that in cases of non responsive recipients, they assumed the recipient was busy. Having ambiguity in a user’s true status enables what Hancock et al. describe as “Butler Lies” [2]. An example of a Butler Lie is ignoring an IM for 30 minutes and then responding to the person saying you were looking at your other screen and missed their IM. Because many CMC tools allow for some ambiguity, users have adopted to create feasible explanations for not communicating. Butler Lies are a way in which deception is used to control availability, and they are beneficial to both parties because it lets them save face. The liar does not appear mean and the person being lied to can still feel like they are respected.

Birnholz et al. “urge designers to weigh the value of more information against the threat to potentially valuable ambiguity. Consider options that allow people to share information at multiple levels of detail (such as what city or neighborhood they are in, but not the specific address) and only with specific contacts.” [5].

I believe with the ability to manually set statuses, there is no further need to provide more detailed information on a user’s “true status.” Having to answer for every ignored IM or deflected call because the sender knows exactly what the recipient is doing only creates a burden on the already busy worker.

Conclusion

There are areas where the tools can better support needs around managing interruptibility and unresponsiveness, but as a whole the tools are utilized in ways that blend well with other activities and job duties. I found that users are sometimes frustrated with their interruptions, but this was not because the tools were insufficient, but because people needed to collaborate and they would still need to interrupt even without IM.

There are several areas that should be further explored. Research should be done on how workers manage interactions with specific people they interact with, particularly with non-peers. I found that there was no expectation that upper management would be available through IM, but did not talk to upper management to research their views and use of IM. I think additional exploration into the non-text features of IM systems is warranted. For instance I found that users frequently cited the screen-sharing feature of Lync as a benefit when collaborating with others.

I found that IM users in the workplace have created a place in which they can effectively balance their individual work with the collaborative work they must also support. Each environment was slightly different, and users have found what works in their case. Users were very aware of their availability and could manage it fairly well to control their interruptions. Knowledge workers are busy and use IM to help support others while also being able to focus on their own work. This often comes down to a situation of priority management and users will never be able to solely rely on a tool to prioritize for them.

References

  1. J. Birnholtz, N. Bi, S. Fussell, Do You See That I See? Effects of Perceived Visibility on Awareness Checking Behavior, Proc. of CHI 2012, (2012)
  2. J. Birnholtz, J. Hancock, M. Smith, L. Reynolds, Understanding Unavailability in a World of Constant Connection, Interactions, Sept – Oct, (2012), 32-35
  3. R. Garrett, J. Danziger, IM=Interruption management? Instant messaging and disruption in the workplace. Journal of Computer-Mediated Communication, 13(1), (2007), article 2. http://jcmc.indiana.edu/vol13/issue1/garrett.html
  4. J. Hancock, J. Birnholtz, N. Bazarova, J. Guillory, J. Perlin, B. Amos, Butler Lies: Awareness, Deception, and Design, Proc. of CHI 2009, (2009), 517-526
  5. M. Handel, J. Herbsleb, What Is Chat Doing in the Workplace?, Proc. of CSCW’02, (2002)
  6. J. Herbsleb, D. Boyer, D. Atkins, M. Handel, T. Finholt, Introducing Instant Messaging and Chat in the Workplace, Proc of CHI 2002, 4 (1), (2002), 171-178
  7. C. Ou, R. Davison, Interactive or interruptive? Instant messaging at work, Decision Support Systems, 52, (2011), 61–72
  8. L. Palen, P. Dourish, Unpacking “Privacy” for a Networked World, Proc. of CHI 2003, (2003)
  9. D. Redmiles, H. Wilensky, K. Kosaka, R. de Paula, What Ideal End Users Teach Us About Collaborative Software, Proc. of GROUP’05, (2005), 260-263
  10. L. Ruge, M. Kindsmüller, J. Cassen, M. Herczeg, Steps towards a System for Inferring the Interruptibility Status of Knowledge Workers, Proc. of ITM’10, (2010), 250-253
  11. J. Wiese, J. Biehl, T. Turner, W. van Melle, A. Girgensohn, Beyond ‘yesterday’s tomorrow’: Towards the design of awareness technologies for the contemporary worker, Proc. of MobileHCI, (2011), 455-464

Use React Native to post to secure AWS API Gateway endpoint

I am setting up a React Native application that will interface with an authenticated API hosted by AWS API Gateway. Here is how I set up my API to be secured through authentication. I am not sure that this will be used in production, but it is working well for testing.

This post will go over the following:

  1. Setting up a very simple React Native application
  2. Adding a simple button that will later be used to get data from an endpoint
  3. Using the react-native-dotenv module for environment set up
  4. Using the react-native-aws-signature module for authorization
  5. Debugging with react-native-aws-signature

Here is the code for this example on githhub

Setting up a very simple React Native application

Start with a brand new react-native application. To set one up, run:

You should get something in the simulator that looks like this:

Adding a simple button that will later be used to get data from an endpoint

In the index.ios.js file, add Button to the imports:

Replace the existing SampleProject Component with this:

Reloading in the simulator should give you something like this:

If you press the ‘API Request’ link, you should get this:

Using the react-native-dotenv module for environment set up

In a production mobile application, you don’t want to save secret API keys anywhere in the code because it can be reverse engineered. There is a SO post here about it.

That being said, if you are only installing the app on your phone during the testing phase, it is probably fine.

The official react-native-dotenv instructions are here, but this is what I did to set it up.

First, install the module

Add the react-native-dotenv preset to your .babelrc file at the project root.

Create a .env file in your project root directory with your AWS credentials and the host.

Now, let’s set up a really simple class that we will use to interface with our API. This should be at the same level as index.ios.js, and mine is called called SampleApi.js.

Then, somewhere near the top of index.ios.js, import the new class:

Replace the retrieveData function with:

Our full index.ios.js should now look like:

Note: if you change the .env file only, the simulator will not recognize the change and your changes will not take affect.

Using the react-native-aws-signature module for authorization

Now, we want to actually hit the API when the button is pressed. Start by installing the react-native-aws-signature module

In SampleApi.js, add the import for AWSSignature:

Remove the contents of the get() method in SampleApi.js and start by setting up some variables based on the .env file:

Next, set up the header and options. These will be used to generate the authorization details and they will be used in the request to the API.

Then, create a new AWSSignature object and call setParams. This will generate the authorization header, which we retrieve in the next bit of code:

Now, retrieve the authorization information and append it to our header.

Finally, make the request to the API using the header we just created. We are expecting json back, and I have included some basic error checking.

Here is what the SampleApi.js file should now look like:

Modify index.ios.js to set the state to include the return value of the request. Since we are getting a json array back, we have to loop through it to make a readable text block:

After you refresh the simulator, you should be able to press the button and receive a screen that looks something like this:

Debugging with react-native-aws-signature

This AWS troubleshooting guide is helpful, but react-native-aws-signature does most of the work for you, so it can be difficult to determine where your mistakes are.

I got this error when I was including the https:// at the beginning of the host parameter in the header. The full error includes what AWS was expecting for the ‘canonical string’ and the ‘string to sign’.

I figured out how to fix the issue by using the getCanonicalString() and getStringToSign() methods.

 

Create secure endpoints for AWS API Gateway

I am building an application that will rely on the AWS API Gateway for a REST API. I want to make sure that other people are not able to read or write data on the endpoints. I will be using IAM authentication, using the steps below:

  1. Set up an example API
  2. Test that the API works without authorization
  3. Enable authorization on your endpoints
  4. Set up a new User in IAM for API requests
  5. Configure your request to use your credentials

Set up an example API

If you already have an API set up, skip this part.

From API Gateway, select “Create API”.


On the next screen select “Example API” and click “Import”.

The UI will then prompt you to “Deploy API”, if it doesn’t, you can select the option from the “Actions” dropdown. You must provide a stage name for the deploy, I just used ‘test’.

After you have deployed the API, you should see a screen like this, that includes a link to the API.

Click on the link and it will bring you to an info page. We will test the endpoint in the next step.

Test that the API works without auth

Now, make sure you can get to your endpoints without authentication. You can test the GET endpoint by appending ‘/pets’ to your url, either in a browser or with an application like Postman.

The browser output will look something like this:

The Postman output will look something like this:

Enable authorization on your endpoints

Now, let’s lock down the API so only we can get to it. In API gateway, select the /pets GET resource:

Then go to the configs for the Method Request and select ‘AWS_IAM’ under the Authorization setting.

In order for the changes to take affect, you have to use the “Deploy API” action under the “Actions” dropdown. You can deploy over an existing stage, or create a new one.

Now when you try to hit the endpoint via the url, you should get this response

Set up new User in IAM for API requests

Go to your IAM setup and add a new group with the following permissions: AmazonAPIGatewayInvokeFullAccess

Set up a new user that is a part of the group you just created. You won’t need to log in as that user, so don’t set up a password.

On the last screen, the credentials will be provided, make sure you capture both the Access Key ID and the Secret access key, the secret key won’t be displayed again.

Configure your request to use your credentials

In order to get to our endpoint, we need to include authorization values in the header. These are calculated at the time of the request to make sure other people cannot just reuse your headers to gain access.

Here are the full instructions from Amazon, but Postman makes it pretty easy.

Under the “Authorization” tab, select “AWS Signature”

You will be taken to this screen where you can enter the configuration:

The configuration includes:

  • AccessKey – This is the Access Key ID that we copied in the previous step, it is the shorter of the keys
  • SecretKey – This is the “Secret Access Key” that was copied in the previous step, it is the longer of the keys. (It should never be shared)
  • AWS Region – This is ‘us-west-2’ for me, but may be different for you
  • Service Name – this should be ‘execute-api’

After you have entered the values, press “Update Request”. Now if you try to access the endpoint, you should get the data as before.

Check out the values that were included in the “Headers” tab. The ‘X-Amz-Date’ and ‘Authorization’ will change with each request and they are what Amazon verifies on their end to ensure you have up to date permissions.

Next Steps

In order to use the API from an application, you will need to systematically add the headers to your requests. I am using react-native-aws-signature for my React Native application.

Using Amazon Web Services to create a serverless web app

Amazon Web Services (AWS) provides resources that can take the place of a traditional webserver. A huge advantage to this approach is removing the need for the developer to maintain servers, and to allow for easy scaling depending upon use.

In my project, I want to set up mobile app that will use AWS as a backend. I plan on using the following services:

  • Lambda – allows you to run arbitrary code without setting up a server
  • Relational Database Service (RDS) – hosted database. I will be using postgresql
  • Step Functions – Connect services together using visual workflows
  • API Gateway – Create API endpoints to allow your backend to be used outside of AWS

Other technology I will be using

  • Node.js – I am new to JavaScript and want to learn more about it
  • PostgreSQL
  • Bookshelf.js/Knex – I want a well supported ORM that supports PostgreSQL. Bookshelf.js seems robust and since it is built on top of Knex, I can always fall back to Knex for unsupported functionality.
  • node-lambda – Make it easier to manage AWS Lambda deployment

Here is an outline of my steps

  • Set up PostgreSQL and Bookshelf.js
  • Set up hosted database using RDS
  • Create simple lambda that reads from RDS
  • Integrate API Gateway with Lambda
  • Connect 2 Lambdas together with Step Functions
  • Integrate API Gateway with Step Functions

Set up Bookshelf.js for Node.js

Intro

I am going to walk through setting up a brand new Node.js project using Bookshelf and PostgreSQL. At the end, we will have a simple Node.js project that can insert and retrieve data from a database.

Here is what this post will cover

  • Set up a new Node.js project
  • Install necessary libraries to interface with PostgreSQL database
  • Set up process to create a database table
  • Use Bookshelf to
    • insert data
    • retrieve data

The code for this post can be found here.

Why Bookshelf.js?

Bookshelf.js is an Object Relational Mapping (ORM) for Node.js that allows developers to more easily interface with a relational database. It allows the dev to focus on higher level functions, rather than low level queries. It takes care of some basic attribute mapping to make data models simpler. It also allows for the same code to be used against different relational databases in case you want to change the underlying database.

I chose Bookshelf.js because it is well maintained, well documented, and it is built on Knex.js, which is a SQL query building framework. I can use Knex for queries that are not easily supported in Bookshelf.

Install necessary software

Node.js

I am using Node.js 4.3.2 because 4.3 is what is supported in AWS Lambda. If you don’t have node installed, follow these instructions.

PostgreSQL

Instructions on how to install PostgreSQL. I use Postico to easily interface with the PostgreSQL.

Before we go on, start the database.

Set up a new Node.js project

Set up the project directory and initialize the project. When you run npm init, it will prompt you to answer some questions, you can use the default values or set your own, at this point they don’t matter too much.

Install the node modules that we will need, the -save option will add the modules to the package.json file.

Here is what each module does

  • bookshelf – the ORM we will be using
  • knex – required by bookshelf, but in the future it may not be included automatically
  • pg – the postgresql module
  • dotenv – a utility that makes dealing with environment variables easy

Here is our current package.json

The dotenv module reads from a .env file in the project directory. This will set the environment variables. Right now we only need to set which environment we are in.

Hello (node) world

Let’s create a simple javascript file named index.js that will show that we have node working.

Now, make sure that it works

Create a database

Let’s create our database, and verify that it exists.

Create table in database

Set up a knexfile.js config file for Knex. This tells Knex how to connect to the database, notice there are entries for development and production. Right now we care most about development.

Now create a migration that will define our table. The resulting XXXXX_create_table.js file is just a blank template that we will fill out. Yours will have a different date and time in the name.

Later, we will be using the test jsonAPI here. To keep it simple, let’s just create a table that replicates the post resource, which has data like this:

Edit the XXXXX_create_table.js file to include the columns we want our table to have. Mine looks like this

This will create a table called ‘articles’.  We have an id field that will automatically increment when we insert new records. The remote_id is the id from the test service, we want to be sure that we don’t have duplicates, so we use the .unique() method. The table.timestamps(); command will add a created_at and updated_at column that are automatically populated when a record is added or updated. I have excluded the user_id that appears in the test API to keep the table simple.

The exports.down function defines what should happen if we rollback the migration, in general this should do the reverse of what the exports.up function does.

Now, run the migration with knex migrate:latest, and make sure the table exists.

Set up Bookshelf

Start with a bookshelf.js file that will allow us to use bookshelf in other files.

Create a model class for the articles table that we can use to interface with the table. By extending the bookshelf.Model, we get functionality that allows us to access the attributes of the table as methods on the Article variable.

Insert data into the table

Edit index.js to create a new entry in the articles table

Run index.js and make sure that it adds data to the table.

Retrieve data from the database

Now, modify index.js to also query the database for the article we just inserted. Below we have added the getInsertedArticle function, commented out the code that just did the insert, and added some new code to insert and then retrieve the record.

Done

That is it. There are many more things we can do with a database, like create relationships, do complex joins, etc., but this is a good starting point.

Here is a link to the code from this post.

Working with servos and an Arduino Nano clone

Over the past few weeks I have been working on building the software to allow my RC Sailboat to sail autonomously. To keep the form factor as small and light as possible, I bought a cheap Arduino Nano clone to use in place of an Arduino Uno.

Nano with rudder and Sail
Nano microcontroller with servos for sail and rudder control

Obstacle 1: drivers

My first obstacle was getting my mac to communicate with the board. The basic issue is that the WCH340G USB interface chip on the clone is different than the FTDI FT232RL used on the authentic Arduino Nano. There are many sites with instructions and links to drivers, but this one had the most up to date driver that was signed. The older drivers were not signed and required me to bypass OSX’s security to allow unsigned drivers. direct link to driver download

Obstacle 2: consistent power

I had some semi-working software running on the Uno that would adjust a sail and rudder servo. When I swapped in the Nano the servos generally worked, but would sometimes move wildly. Since the Nano and Uno are very similar (chip, ram, clock speed, etc.) I figured the issue was going to be with the power or maybe the WCH340G USB chip.

To troubleshoot I used my voltmeter to see how the Nano voltage changed while adjusting the servos. When not moving the servos, it put out 4.8V on the 5V pin and while moving the servos simultaneously it would drop to ~4.2V. The servos expect 4.8-6V so this could contribute to the issue. As reference, the Uno put out 5V, I didn’t test it while moving servos.

In the past I have had issues with macs not putting out enough voltage through USB, and I have a cord that includes 2 plugs, one for power and one for power+data. The cord is similar to this one.

I swapped out the cords and the servos started behaving as expected. My guess is that a combination of data transfers (to signal the servo change) and keeping a consistent power supply were too much for the single cord.

Since I want to run the system without my computer, I think the USB cord is not the best solution. A couple of alternative options are a separate power supply for the servos, or using a capacitor to smooth out the power.

To keep down weight and complexity, I took the capacitor approach. I found many sources that said a minimum of 470uF was the correct capacity and that the voltage rating should be 3x what you have in the system. I added a single 1000uF/25V capacitor with the old cord and the servos behaved well. The servos also worked well with three 100uF/16V capacitors in parallel, but I prefer the single capacitor solution.

Building nightstands

Introduction

I designed matching nightstands, this page shows how I built them.

Pair of finished nightstands
Pair of finished nightstands

Finished nightstand

Lumber

I wanted dark wood that was inexpensive and easy to work with. I went to Crosscut Hardwoods in Seattle and it was amazing. The staff was super helpful and directed me to roasted poplar. Poplar is normally a blonde color, but they put it into a kiln and it turns a dark brown throughout. Nearly all of the water was removed in the roasting process and then the lumber was cut, so there is very little warping. Also, it has a nice campfire smell.

The lumber was 8-10 inches wide and a full inch thick, which is thicker than I was looking for, but something remedied with a planer.

I bought jatoba (Brazilian Cherry) for the sides of the drawers, mainly because it was already a 1/2 inch thick and I liked the color contrast with the roasted poplar.

I bought 1/8 in. mahagony veneer plywood for the bottoms of the drawers.

Roasted poplar
Roasted poplar

Construction

I spent a full weekend with my dad in his shop creating the nightstands.

Edge Joining

We had to edge join many pieces, but we wanted to get them to the basic length and thickness first. We used a 12 in. planer to bring the thickness from 1 in. to 3/4 in. This was a lot of work.

We drilled holes in the edges to receive dowels, then glued the pieces together and let sit over night.

Doweling jig ready for drilling.
Doweling jig ready for drilling.
Edge joined pieces right after gluing
Edge joined pieces right after gluing
Edge joined piece with the glue wiped off.
Edge joined piece with the glue wiped off.

After the glue was set, we cut the pieces to the exact dimensions. We used a thickness sander to smooth out imperfections of the joint, remove glue residue and get the pieces to the desired thickness.

Drawers

Although I had designed the drawers on paper, I wanted to be sure we had a design that would work with the runner, so I created a wood prototype.

Figuring out how the runners and drawers will work together.
Figuring out how the runners and drawers would work together.

Once I had a good idea of how the runners would work, we attached them to the sides of the stands. There was a lot of trial and error to get them even.

We used self drilling screws to assemble the drawers, predrilling the jatoba because it is such a hard wood.

Finished drawer
Finished drawer
Back of faceplate
Back of faceplate
Back of drawer, recessed to allow for cords to go behind the drawers.
Back of drawer, recessed to allow for cords to go behind the drawers.

Dados

We cut dados for the shelves and the backs of the nightstands.

Dado through a dowel, lucky I guess.
Dado through a dowel, lucky I guess.

Side with dados

Finishing

Prepping for sanding

There were several gaps and imperfections that needed to be filled in, most were small enough that a mixture of sawdust and glue worked well. There were a few gaps that I filled first with slivers of wood and then finished with the sawdust and glue mixture. When mixing, I went for as little glue as possible, while still getting a tacky feel.

I used too much putty in a few places that required a lot more sanding.

Putty made from mixture of sawdust and wood glue.
Putty made from mixture of sawdust and wood glue.
A larger gap that needed to be filled with both slivers of wood and putty.
A larger gap that needed to be filled with both slivers of wood and putty.
Same gap, filled with slivers and putty.

Smoothing and Sanding

I used a card scraper to get the surfaces flush with each other, this worked great and was much faster than sanding. I then used an orbital sander with 150 grit sandpaper. I didn’t spend a lot of time sanding the inside faces.

Polycrylic

I used Minwax Polycrylic waterbased clear matte finish. This was really low odor and dried fast. I did 3 coats, sanding with 220 by hand between coats. Since the drawer faces and top were going to be the most visible, I did an extra couple of coats and sanded with 320 between the final coats.

Before finish on right, after finish on left
First coat of polycrylic
First coat of polycrylic

Finished nightstand

Pair of finished nightstands
Pair of finished nightstands

Finished nightstand

Finished nightstand

Designing nightstands

We were looking for nightstands that wouldn’t take up too much room and would still be functional. We found that many of the options were either ugly, too big, too expensive, or a combination of all three. I decided that I could design and build nightstands that would meet our needs and it would be a fun project.

Requirements

  • Dark wood
  • 12 inches wide
  • A drawer or two to reduce clutter
  • A shelf for cell phones

Planning

Inspiration

I found a few designs that were similar to what we were looking for, here is my pinterest board with some inspiration.

Sketching

I started with some simple sketches to figure out how the proportions would work and to see if two drawers were too many.

From the inspiration night stands and through my sketches, I realized that I would prefer to have drawers without pulls, to give a sleek look.

Designing

I spent some time in Illustrator laying out several options. Since neither of us have many large books and we have a separate bookshelf, we decided that a shorter bottom shelf was a good compromise for 2 drawers.

Possible designs for nightstand
Possible designs for nightstand, including pull cutouts. (I went with E)

(Almost) final plans

Nightstand Views

The pieces that are required for the nightstand
The pieces that are required for the nightstand

The grid squares are 1/2 inch. I assumed 3/4 in. thick boards, except for on the drawers. Most of the faces are wider than regular lumber, so I assumed we would have to edge join pieces to make boards that were wide enough.

The drawer front pieces, which are just behind the faceplate are a bit shorter to make sure your hand doesn’t hit them when you pull the drawer open.

The gap behind the drawer was originally created to allow a cord to go behind the drawers, but I instead opted for drilling a hole in the back of the nightstand.

The drawer bottoms are 1/8 in. plywood.

I used Ikea Besta drawer runner slides. The plans are missing the details on where the slides are positioned. This was something we figured out at build time.

Prototyping

To make sure the nightstand would be an appropriate size, I created a very rough prototype out of cardboard. I was happy with the size and dimensions and was ready to start the build process.

Cardboard prototype

The build process: Building nightstands

How to display Google Earth maps on website

When I hike, I track my routes with a GPS and import them into Google Earth. This works great on my personal computer, but it doesn’t allow me to easily embed the maps on a webpage. To allow others to view my maps, I import them into Google Maps and then embed these custom maps on my webpage.

Steps to embed Google Earth routes on webpage using Google Maps

  1. In Google Earth, right click on the route you want to save and select “save place as”save place as
  2. Save the file as a .kml
  3. Sign in to Google Maps and open the menu on the left side of the search bar by clicking the three line icon
    menu
  4. Within the menu, select “Your Places”, then the “Maps” tab.
    your places
    Maps tab
  5. This page will show all of your custom maps, click the “Create Map” link at the bottom of the page
  6. The map edit screen will open with one map layer created for you. Click “import” under the layer. This will let you select the .kml file that you created earlier.
    import
  7. If you want to add multiple routes to the same map (for instance a multi-day hike), add a new layer with the “Add layer” link, and import a separate .kml file.
  8. You can change the look of the map. These changes will be reflected on the embedded map if you change them in the future.
    • To change the color of the route, mouse over the name at the bottom of the layer section (“Enchantments Through hike” in the images below) and click the paint bucket icon.
      color
    • To add a label to the route, click on the “Individual styles” link and select the label you want to show.
      label
    • You can change the type of map (terrain, satellite, etc.) from the “Base Map” link under the layers section.
  9. In order to embed the map, you will have to make the map public through the “share” link.
    public
  10. Once the map is public, you can select the “Embed on my site” item from the menu.
    embed on my site
  11. Insert the provided code on your site, and you are done.
    copy code

 

Here is an example of an embedded map

Viking burial

A viking burial for a frog

Why I am building the boat: Balsa viking boat

How I made the plans: Viking boat plans

Boat construction: Viking boat construction

Preparation

We wanted the boat to sink when it was done burning, so we added some rocks in the compartments at the bow and stern.

We also wanted to be sure Freddie burned before the boat sank, so we added a bed of wood shavings and kindling around the internal deck. We also used long pieces of cedar to look like oars raised in the sky.

We attached a wire loop to each end of the boat and then tied fishing line to the loops. This way we could easily keep the boat centered in the pond.

Adding Freddie

Freddie had spent the last several decades in a jar of formaldehyde and had pretty much decomposed into a cloudy solution. There were still some solid parts, so we created a net of cheesecloth with a wire hanger and poured him into it. We rinsed him off with alcohol to make him more flammable, wrapped him in the cheesecloth to create a shroud, and placed him in the boat.

Too long in a jar
Too long in a jar

The sending off

My mom created a sound track for the funeral which included Turn! Turn! Turn! by The Byrds and Joy to the World by Three Dog Night.

We gathered around the pond, started the music and then lit the boat on fire. It burned quite well and by the second round of Joy to the World, the boat had burned down to the bottom of the hull, turned over, and sank bottom up to bottom of the pond.

Boat at the bottom of the pond