We recently had the opportunity to build a collaborative whiteboard for a client, and we were surprised when we found out how easy its implementation was using PencilKit and Twilio Data Track. The idea is to implement a whiteboard that users can access from multiple devices. When any of the connected users draws on it, the drawing will be immediately displayed for the other users to see.
Our goal with this article is to show you how to accomplish this with these two awesome tools but, before we get down to work, we will need a little bit of background theory 🤓
Tools
PencilKit
PencilKit provides a drawing environment for an iOS app that receives input from Apple Pencil or the user's finger and turns it into images displayed in iPadOS, iOS, or macOS. Some components you'll need to be familiar with in order to start using PencilKit are: PKCanvasView, PKDrawing and PKToolPicker. In this post we will keep the focus on the implementation of the whiteboard, but feel free to refer to the Apple official documentation to read more about them in case you need it 😊.
Twilio Data Track
Twilio Data Track is the mechanism/gateway that we'll use to send the drawings, that the local participant has drawn, to the other users in order to share them in real time. The Data Track API lets you create a Data Track channel which can be used to send low latency messages to zero or more receivers subscribed to the data. In this example the messages are the drawings.
Note: Twilio offers two types of tools to send data in real time: Twilio Sync library and Twilio Data Track. The main difference between using Twilio Sync library and Twilio Data Track is persistence. Data Tracks only exist during the video meeting as the shared data among the participants is not stored, while Twilio Sync documents can be stored under an account, get an identifier and retrieved for new updates.
For this particular use case, the best option is Data Tracks as the whiteboard data (drawings) would not be required to be saved. To learn more about Data Tracks please refer to the Twilio official doc.
Now that we are familiar with the foundations..
Let's get down to work!
Twilio Data Track is a subcomponent of the Twilio Video library so we need to add Twilio Video library in the podfile in order to use Twilio Data Track.
Code
Roughly speaking, the steps to implement the whiteboard are:
- Setup Twilio Data Tracks and Connect to a Twilio room.
- Setup PencilKit.
- Draw and send the drawings over Data Tracks.
- Configure Twilio Data Tracks delegates to receive the drawings and set them on the canvas.
As we mentioned earlier, you'll need to be connected to a room and of course have PencilKit set up before going any further. We have built a sample project in which you can review how to do this. In this article we will only focus on points 3 and 4.
Draw and send the drawings through data tracks
After setting up PencilKit you should be able to draw (yes - it’s that simple). Now, the thing is, how are we going to manage to capture that drawing in order to be able to send it using LocalData Tracks? The answer is quite simple: we capture the drawing through the delegate method canvasViewDrawingDidChange, as shown in the following snippet:
canvasViewDrawingDidChange (belongs to the PKCanvasViewDelegate) tells the delegate that the contents of the current drawing changed. In other words, this method will be triggered once the user lifts his finger or pencil. It is at that moment that we send the drawing as a base64 string in a dictionary via Data Track.
Configure Twilio Data Tracks delegates to receive the drawings and setting it on the canvas
Listening for RemoteDataTrack events
The Twilio RemoteParticipant class provides a delegate protocol named RemoteParticipantDelegate.
Receiving Messages
You need to implement RemoteDataTrackDelegate to receive incoming messages on a DataTrack.
Note: The DataTrack API supports sending string as well as byte data. You can use sendString or sendData to send data to the Room. In this case we are sending the drawings as a jsonString, that’s why we implemented the remoteDataTrackDidReceiveString.
Need of needsToStoreOnDB flag
Issue 👉 We noticed that, after implementing the remoteDataTrackDidReceiveString method, we ran into a circular reference problem. After a little while of digging into the issue we noticed that anytime the local participant would draw on the canvas, the method canvasViewDrawingDidChange was called. This method sends the drawing through the DataTrack. After sending it, since the local participant is listening to the DataTracks events as well, the remoteDataTrackDidReceiveString method is invoked which also sets the received data (drawing) on the canvas by doing canvasView.drawing = pkDrawing. This line of code causes the canvasViewDrawingDidChange method to be executed (again), starting the cycle all over again.
Solution 👉 To fix this issue needsToStoreOnDB flag comes into action. It indicates whether the drawing should be sent through the DataTrack; we only send the drawing if it was drawn by the local participant. If it is a drawing that is coming from a remote participant we do not send it, only display it on the canvas.
And set needToStoreOnDB back to false:
Run the app to see the final result..
You can find our Whiteboarding implementation here so you can try it! 😻
Extra/Disclaimer:
To test the project out you need to generate a Twilio access token and replace it on the accessToken variable on the WhiteboardViewController. Since each token is unique for each participant, if you want to test it for two users (participants) you need to generate one access token, replace it on the code and run the app in one simulator, generate another access token and run the app again on a different simulator (or device). Don’t forget to change the Name (CLIENT IDENTITY) on the Twilio Console and set a cool name for your room before generating the access token.
