Arduino Light Theremin (Starter Kit Project #6)

We will use a phototransistor as a light sensor to control a piezo and produce sound – resembling the functionality of an actual theremin.

Setup

Here’s what we need:

  • Arduino UNO + Breadboard
  • Jumper wires/cables
  • 1 10kΩ resistor
  • Piezo
  • 1 phototransistor

Layout

Here’s what the instrument should look like, as per the Arduino projects book.

And here is what I made:

Code

The calibration of the sensor is the most important part in this mini-project.

int sensLow = 1023;
int sensHigh = 0;

void setup() {
    while (millis() < 5000) {
        sensVal = analogRead(A0);
        if (sensVal > sensHigh) {
            sensHigh = sensVal;
        }
        if (sensVal < sensLow) {
            sensLow = sensVal;
        }
    }
}

The millis function returns the amount of time for which our board has been working. Thus, we calibrate for the first five seconds by taking in the minimum and maximum possible sensor input values. These will be used to scale our input values later on.

void loop() {
  sensVal = analogRead(sensPin);
  int pitch = map(sensVal, sensLow, sensHigh, 50, 4000);
  tone(8, pitch, 20);
  delay(10);
}

Here, we take in our input value and map it to the appropriate frequency. Then, using the tone function, we output sound using our piezo (which vibrates as per the frequency provided by tone()). You can play around the frequency and mapping values just for fun.

Arduino Mood Cue (Starter Kit Project #5)

Here, we will turn a potentiometer to consequently rotate a simple servo meter. The arduino will collect the data a

Setup

Here are the components we need:

  • Arduino UNO + Breadboard
  • 2 100 μF Capacitors
  • Jumper wires/cables
  • Potentiometer
  • Servo Motor (with arm)

Layout

Here’s what it should look like, as per the Arduino projects book.

And here is my build:

You might notice that there are some differences between the schematic and what I’ve made. Firstly, the potentiometer is just too large to be placed on the same strip as a capacitor and power/ground connections, which is why I have used orange jumpers to elongate the entire setup. Also note that the capacitors, if connected incorrectly, could potentially explode. I would google up how to connect these and then proceed! It took me a lot of time to find out which pin is the anode and which pin is the cathode. Turns out the pin which has a grey arrow full of zeros pointing downwards is the negatively charged electrode, or the cathode.

Code

You can find the full code here.

#include <Servo.h>
Servo myServo;

To start, we import the library required to interact with the servo motor, and create an instance of the Servo object to handle our motor. Then, all we need is 4 lines of functional code.

void loop() {
    potVal = analogRead(potPin);
    angle = map(potVal, 0, 1023, 0, 179);
    myServo.write(angle);
    delay(15);
}

Firstly, we read our analogue potentiometer value and convert it into an angle for rotating our servo motor. Then, using the servo library, we turn our motor. The delay of 15 milliseconds is to accomodate the time used up in rotating the motor.

Demo

Arduino Color Mixing Lamp (Starter Kit Project #4)

This entire thing is fairly simple. We will use phototransistors to input the RGB components of any light source to our arduino. The arduino will then convert these to digital values, and combine the RGB components to operate a four pin, RGB LED. All we have to do is input to our uno, process the data (fancy word for dividing by 4), and output.

Setup

Here’s what we’ll need:

  • Arduino UNO
  • Breadboard
  • 3 220 Ω resistors, 3 10 kΩ resistors.
  • Jumper wires/cables
  • 4 pin RGB LED
  • RGB color filters
  • 3 phototransistors

Layout

Here’s what the circuit should look like (from the arduino starter kit projects book).

And here is my build:

You can see that I didn’t use the wooden components already in the starter kit to appropriately connect the filters to the phototransistors. This is because those components were too small (probably a defect in the kit), fitting the screens caused them to fly across the room at random intervals.

Code

You can find the full code here. I’m not discussing the setup part here.

void loop() {
    redSensVal = analogRead(redSensPin);
    delay(5);
    greenSensVal = analogRead(greenSensPin);
    delay(5);
    blueSensVal = analogRead(blueSensPin);
}

Here we’re reading in the rgb components of the light source under scrutiny. A delay of five milliseconds has been taken as the arduino takes a while to process analogue inputs and convert them into digital values.

void loop() {
    // ...
    
    redVal = redSensVal/4;
    greenVal = greenSensVal/4;
    blueVal = blueSensVal/4;

    analogWrite(redLEDPin, redVal);
    analogWrite(blueLEDPin, blueVal);
    analogWrite(greenLEDPin, greenVal);
}

Now we divide by four because our analogue input corresponds to digital values between 0 and 1023. However, our output LED requires digital values between 0 and 255. Given that our output LED requires analogue inputs, we use the UNO’s built in DAC to display our combined RGB components.

Note

Many people (including me) initially thought that this entire thing wasn’t working. This isn’t the case. The fact is that the phototransistors that come with the starter kit are incredibly insensitive. They registers values of o,0,0 for room lights (and most lights), which is why there is no discernible output unless one uses a strong flashlight (built-in in smartphones) or a decent light source.

Here’s a small demo. You can visibly see that iphone flashlights have strong blue and green components.

Mini-Demo

Summary of Reading – July 19′

Breakout Nations; Ruchr Sharma; 2012; non-fiction – A fascinating book that is made even more interesting given how it gives a somewhat unique insight into what the world was forecasted to look like almost a decade ago. This is wonderful given how some of the long-term predictions Sharma made in his book are visibly underway, whereas some have been woefully torn apart by economics, politics, and war. Lastly, I especially admired the section on the Indian subcontinent, specifically the portion where Sharma gives his views on the ‘rising’ Rahul Gandhi. A truly wonderful read that links together the past and the present.

Prisoners of Geography; Tim Marshall; 2015; non-fiction – Another brilliant book that managed to convince me that a field of study I previously considered alien and declining plays a much larger role in our world – geopolitics. Marshall successfully navigates the issues of explaining politics using nature. However, on the downside, his language is rugged and not wholly engaging – in stark contrast with the ideas he attempts to portray. There was also too much negativity buried in how most world maps were drawn by ignorant leaders (although I do not know whether it is right to question that).

The Soul of an Octopus; Sy Montgomery; 2015; non-fiction – A fascinating and complex book that enthralled me right from the start. As a presumption, I assumed that, even before starting to read it, that writing about octopuses would be incredibly hard. However, the author manages to weave together some alien facts into a cohesive structure. She also fuses together the best of fiction and non-fiction elements, giving the book a spectral quality. In fact, for a certain period of time, one feels galvanised to make his/her own aquarium. However, what I found lacking was the author’s focus (especially in the latter parts of the book) on worthless stuff that vitiates the flow of language and contradicts her purposes at the beginning. One such example is the chapter on getting octopuses to blind date, which makes no sense to me (given the context).

Contact; Carl Sagan; 1985 – Sagan, once again, doesn’t fail to impress. What I admired about this book was the constant sense of longing, longing to conquer what lies beyond our pale blue dot. Combined with the conflict between humanism and politics which dominates the plot, and the variety of character traits Sagan imbibes into his characters, the book is a stunning read. From people like Kitz, who is frozen in the Cold-War and thus elects to ignore fact and choose hypocrisy, to Hadden, the very definition of a mad scientist x tech wiz, the book contains a wide array of characters differentiated by not just nationality and ideology, but by opinions, thoughts and individuality. Although I fail to agree with Sagan’s final impression of love (cliché) being the only common denominator amongst ‘The Five,’ this book did leave a lasting effect – a will to wonder about whether or not such advanced civilisations actually do exist out there. I completely sympathised with Ellie’s character, and especially her stubbornness and arrogance when it came to denouncing opponents logically. Hadden too had a deep impact on me. His solicitude combined with his love for space and inability to stop believing in the most arcane theories possible made him a role model esque figure. Although his death came a bit too early, I couldn’t help but relate to his desire to see Jupiter’s swirling mass of hydrogen and helium. On the other hand, although I didn’t like it, but one has to accept that Sagan’s repeated emphasis on flawed human characteristics such as political borders transcending into science, inability to believe in the truth and willingness to accept religion did pay off by representing some of the biggest challenges we face as a species. This book will surely leave you shaken, and perhaps cause many to see how important space truly is and how fragile our planet is.


Currently Reading

Automate the Boring Stuff With Python; Al Sweigart; 2015

Programming in Python3; Mark Summerfield; 2015

The Glass Palace; Amitav Ghosh; 2016

SAT – Starting Prep

This is going to be a really short post. I took my first ‘full practice test’ on Khan Academy’s official SAT practice page today. Long story short, I had been doing random practice questions for a few days so I thought I would nail it, and was naively aiming for a 1500+, maybe even a 1540/60. Unfortunately, I got 1430.

So yeah, this is more of a ‘challenge accepted’ thing. I’ve decided to practice hard every day and take a full test every Sunday as a status update.

Furthermore, I’ve been going through the SAT subreddit, and have done some research, and it turns out that Khan Academy’s 3rd full-test is really tough. So, I’ll take the long term approach (which is rare) and do the 3rd test at the end of August, and basically just give it my all until then!

I’m hoping to get to a 1500 next week. My math was pretty strong (two mistakes, one silly). I want to get a full on both the math sections next week. English needs a lot of work. I’m aiming for a 720 in english for next week. Adios!.

Extracting Passwords Using Regular Expressions in Python.

This is my second post on regular expressions. Here, I’m attempting to solve one of the practice tasks given at the end of Chapter 7, Automate The Boring Stuff. Instead of merely identifying whether or not a given string is a strong password, I will write a program that takes a list of strings, and returns a list containing those strings which will make valid ‘strong passwords.’

  • Extract potential strong passwords from text in Clipboard
  • Strong password must have both uppercase and lowercase characters, as well as digits. It must also be at least eight characters long, and must not have any spaces.

So for this piece of code, we will be using ‘lookaheads,’ a new concept for me pertaining to the usage of regexes. Firstly, here is some basic information about ‘lookaheads.’ A ‘lookahead’ is like an if condition. It follows the following format, (note that not many websites refer to it in this way, but I find this intuitive) (?=Regex1)(Regex2). Here, Regex2 is considered if and only if (iff) Regex1 is true. Now, this means that like in an if ... else conditional statement, we can write code that could mean: (?=if digit is present)(read string). Also, do note that Regex1 being true doesn’t affect how Regex2 is read. Another thing we need to do is ensure that the we scan only one password, and not the entire text.

We can do this by either specifying all possible characters except spaces inside square brackets as follows. [a-zA-Z0-9_+$%...]*. Or by simply specifying ‘all non-space characters,’ which is much simpler.[^\s]*. One could also use \w*, but this would not scan characters used commonly in passwords, like %, #, @, ! et cetera.

Then, we need three lookbacks. One to check for a lowercase character. One to check for an uppercase character. One to check for a digit. We also need to use {} brackets to ensure that the password is a least 8 characters long. Here is the code:

strongPassword = re.compile(r'''(
    (?=.*\d) 
    (?=.*[a-z])
    (?=.*[A-Z])
    [^\s]{8,}
    )''', re.VERBOSE) 

res = []                            
for potentialPassword in strongPassword.findall(text):
    res.append(potentialPassword)                        
    print(potentialPassword)
    

The easiest one is the last statement, which says accept all non-space containing strings that follow the lookaheads above and are at least 8 characters long. Each lookahead follows the same format.

(?=.*\d), which is equivalent to (?=.*[0-9]) says match any character that is a digit. Similarly, the other two regex components say match any character that is a lowercase alphabet and uppercase alphabet respectively.

We then just output our results.

You can find the full code here at Github.

Personally, I found this really difficult mainly because I didn’t know how to chain multiple lookaheads together. Simply putting them one after the other without the .* required the characters to be ordered (lowercase first etc.). Nesting them like a bad if..else statement provided similar results. I ultimately found the correct solution here on stackoverflow, through an implementation for jquery, although not without surfing for over 1.5 hours.

Moreover, the implementation in the official regex website was absolutely nightmarish. A long, single line that was overflowing onto the right side of their webpage. Turns out that their implementation doesn’t even work, for reasons I couldn’t comprehend.

Using Reg. Expressions to Extract Useful Information in Python.

Having just completed chapter 7 of Automate the Boring Stuff. I wanted to test my skills. I’m writing down some code that will carry out the following tasks.

  • Use text copied onto the clipboard
  • Go through text and store, or save URLs, email addresses, phone numbers and ZIP codes.
  • Paste the information found into the clipboard, and print on the CLI as well.

The full code is available here on Github

Firstly, to do this, we’ll need two modules. The re module for writing and using regular expressions, as well as the pyperclip module for using our clipboard. Firstly, let’s import these modules. and write down the some rough formats for the stuff we want.

#! python3
import pyperclip, re
# These are our formats.
#
#   PHONE         | EMAIL           | ZIP      | URL
#   --------------|-----------------|----------|------------
#   area code*    | username        | 6 digits | protocol
#   separator*    | @               |          | server
#   1st 3 digits  | domain          |          | file name
#   separator     | .(com)          |          |
#   last 3 digits |                 |          |
#   extension*    |                 |          |
#
#   *optional

Now that we have this information, let’s make a regular expression, or regex for short for each category.

For phone numbers, we want three optional items and three necessary items, as shown above. To make items optional, we can group them using brackets and then place a question mark so that the number is read even if the contents of the bracket are not present. Like this: (<regex>)?. The ? operator specifies a group that either occur once or not at all. Now, let’s write our regex for phone numbers. Do note that I will use re.VERBOSE to spread the regex over multiple lines, for ease of readability.

phoneRegex = re.compile(r'''(
    (\d{3}|\(\d{3}\))?
    (\s|-|\.)?
    (\d{3})
    (\s|-|\.)
    (\d{4})
    (\s*(ext|x|ext.)\s*(\d{2,5}))?
    )''', re.VERBOSE)

Let’s go over this line by line. The first line, (\d{3} | \(\d{3}\))?, is for scanning any area/region/country codes. We’re assuming these codes to be 3 digits in length. The | is added to ensure that we can read codes regardless of whether or not they are enclosed within brackets. Some possible extensions are 761 or (342). If there is a code, there will be a separator between it and the rest of the number, which is why the second line is required. We are allow for either a space, or a hyphen or a dot. For example, these numbers (with codes) would be read: 342-345-5454, 342.345-5454, and (342) 345-5454. From then onwards, the regex is pretty simple. The third line just scans three consecutive digits (as is mandatory in phone numbers). The fourth line is just a repetition of the second, as elements within a number are always separated by a hyphen or a space in most cases.

The last line could be a bit tricky. It is meant to include any extensions that the owner of the phone number has. \s* is added as there can an arbitrary amount of space between the number and extension. Then, ext|x\ext., where . is the wildcard character accounts for the way extensions are denoted. Lastly \d{2,5}accounts for extensions that are 2-5 characters long. Here are some sample phone numbers with extensions: 345-6571 x 453, (423) 341-9872 ext 45, and 221-986-1034 ext*86298.

Now, time to write down our regex for email addresses.

emailRegex = re.compile(r'''(
    [a-zA-Z0-9._%+-]+
    @
    [a-zA-Z.-_]+
    (\.[a-zA-Z]{2,4})
    )''', re.VERBOSE)
    

[a-zA-Z0-9._%+-]+ represents an expression that occurs at least once (may occur more than once but no 0 times). The content of the square brackets represents our own class of characters. a-z A-Z 0-9 represents any alphanumeric character. Email addresses may also contain periods, underscores, % signs, + signs or hyphens (terribly common), which is why they are included. Then we have the @ sign. After that, we need to identify the domain, which could be made up of any alphanumeric character along with hyphens, underscores and periods. All of these are included inside our square brackets. Lastly, \. represents a period, to start .com or .net. This period is followed by a string of alphabets that is 2 to 4 digits long. Examples – .nl, .in, .com, .gov.

The ZIP code is really easy. We just need six consecutive digits.

zipRegex = re.compile(r'\d{6}')

The URL regex is also easy, we just need to find all ‘words’ startin with http, which covers https in its own, and ending with a .<something>.

urlRegex = re.compile(r'http.*?\.[a-zA-Z]{2,4}')

Using the pyperclip module, we will copy our text from the clipboard. Although this may vary from person to person, I want to include all search results in one list instead of different lists for different types of searches. Here is the code.

text = str(pyperclip.paste())

# <...
#       various Regexes
# ...>

res = []

for number in phoneRegex.findall(text):
    phoneNum = '-'.join([number[1], number[3], number[5]])
    if number[8] != '':
        phoneNum += ' x' + number[8]
    res.append(phoneNum)

for email in emailRegex.findall(text):
    res.append(email[0])

for url in urlRegex.findall(text):
    res.append(url)

for ZIP in zipRegex.findall(text):
    res.append(ZIP)

The functionality pertaining to searching for and storing a valid phone number is complex as our output changes with whether or not the phone number contains the area code and/or extension. We then copy all information gathered to the Clipboard, and print ‘no information found’ if the list of results is empty.

if len(res) > 0:
    pyperclip.copy('\n'.join(res))
    print('Copied to clipboard...')
else:
    print('no information found')

The entire code can be found here. Also, there are some helpful links for understanding Regexes below.

There is an issue with WordPress, links randomly open in new tabs or the same tab, so be careful.

Insights from ‘The Soul of an Octopus’ (Sy Montgomery)

It would suffice to say that this book was an eye-opener. It forced me into recognising octopuses as sentient and heterogenous beings. Some hints were even made, quite convincingly, to suggest that they are smarter than us. In short, I don’t often encounter instances wherein humankind’s intellectual credibility is questioned. Montgomery’s striking language further elucidates upon the emotions, facts and narratives she conveys. Some of these are mesmerising by themselves, such as her emphasis on how octopuses, like humans, savour food before consuming it. Another fact, more daunting than inspiring, is that humans are like twigs to octopuses. Mere dolls in the face of the monstrous forces these marine beings can generate. All in all, The Soul of An Octopus is both frightening and revealing. Frightening in that it makes octopuses seem to be creatures favoured by evolution. Revealing in that it removes the veil that makes us humans believe ourselves to be godly with respect to the planet.

To not see how cunning octopuses, one would have to be more than blind. For example, the tale of Octavia. Octavia managed to simultaneously greet five people with five limbs, while using her remaining three limbs to steal a bucket of fish. Genius. Accompanying such stories is some delightful language. Montgomery, like any good author, makes us feel as if the events are happening right in front of our eyes! She gives every tale a real-world link. The loss of a beloved one – the demise of an octopus. The horror of Alzheimer’s – the nightmare of Octopus’s senescence. Humans solving riddles – Octopuses pening boxes and operating levers to get to their food.

On the downside, the palpable sense of magic turned into boredom and frustration due to how the author attempted to force the readers into mirroring her feelings, especially towards the latter stages of the book. For example, near the end, Montgomery, chose to dedicate 20 odd pages to an octopus blind date. This made no sense to me. As the book approached its end, it became evident that Montgomery wasn’t trying to broach a topic, but wanted to transform the reader into octopus-fanatic overnight. Dull.

Having conflicted good and bad, let’s move on.

After I finished reading, there was one thing pertinently nagging at me – consciousness. The idea of consciousness, in broadly speaking all animals, was evoked again and again. Luckily, this paid off. Do correct me if I’m wrong, but the very idea, or rather concept of consciousness is rather ubiquitous. We don’t understand what it is. Not by any means. As exhibited in The Soul of an Octopus, even organisms lacking neural systems have displayed individualistic behaviours (I’m talking about the starfish). This idea of ‘consciousness bereft of a brain’ may seem absurd, but it is true. We have always thought ourselves to be distinct based on consciousness. Turns out, we’re not that different when it comes to being sentient.

Research teams around the globe have found evidence of both thought and curiosity in many different types of organisms. The octopus is one of many. Not only have octopuses exhibited traits of intelligence, but have shown curiosity by exploring new areas (eg. new aquarium tanks), using various types of solids as armour (eg. coconut shells) and interacting with humans. The most understandable of these examples is that of Octavia, who remembered her handler’s warming presence not by sight, but by touch! And then proceeded to envelop him in octopus slime, while ignoring her food.

It is hard to say if we will ever understand what consciousness is. Despite this, one thing is certain, we aren’t the only sentient organisms on this pale blue dot.

Stockholm – Some Talking Points

Biking

From a biker’s perspective, Stockholm is nothing short of a paradise. In over two weeks, I’ve stumbled across a vast bike-road network consisting of silky tracks. In fact, to my surprise, I’ve quite often found ways through the city that are shorter than those a car would take. Having pedalled through forests, farm-fields and an urban jungle – I would definitely recommend buying/renting a bike. Some good stores are decathlon and bikester (for cheap options). You can also find some decent second hand bikes on “www.blocket.se.”

One potential downside, especially for those who hail from flat terrained areas is the Swedish landscape. It rolls up and down like waves frozen on the surface of an ocean. This makes biking in Stockholm both mesmerising and tough. The struggle of climbing up steep ascents replaces the ecstacy of flying down hills every minute. On a positive note, Biking here is rewarding given that it teaches how to use gears. Back where I come from, Jaipur, gears are pretty much useless given that all roads are completely flat. Here, gears are more important than velocity. It is crucial to change to the right gear at the right time.

Grimsta Natural Reserve

Personally, I love the Grimsta area, and also the route from Hasselby to Bromma Airport. The latter is a clear-cut forest trail, full of jagged rocks, slippery declines and unfathomable heights at times. Last week, while biking on this trail, I even encountered a deer (it bolted away upon hearing me). I did miss out on the opportunity of bird-watching here, so do bring your binoculars if you can.

The trail is, at it its best, completely cut off from the surrounding residential areas. Although Google Maps will guide you through it, one of the best way to actually explore here would be to use your natural compass. A definite vote of approval from me, and a place explorers and bikers can’t afford to miss.

Hasselby –> Bromma

The journey to Bromma from Hasselby isn’t as isolated as the Grimsta trail. One gets to cross small streams, rustic wooden bridges and if fortunate, experience an airplane fly overhead before crossing Bromma airport. You can feel the wind surge it generates! If you have the time, you can even walk while surrounded by acres of farmland with almost no one in sight.

Landmarks

Stadsbiblioteket – Every website will you warn you about missing out on this stunning library. I spent my entire saturday in here, and I can assure, a trip here wouldn’t be wasteful. You can get the descriptions on other websites, but be sure to avoid sticking to the main hall. The library has more than five smaller halls, full of books in a vast array of languages. Furthermore, you can also get free water and wi-fi here, along with a station for charging your electrical appliances.

Armé Museum – Haven’t fully seen this one yet, but the gallery themed on the 16th to 19th centuries is amazing. Seldom will you find a museum where the number of life-scale models and touchable stuff (yes, I’m not lying) outnumbers items hidden away behind glass walls. Seldom will you find a museum with so many to-scale models. Also note that it has quite a high number of items that can be touched and lifted. Like many other museums in Stockholm, this too is free to visit (and also gives free lockers and charging stations).

Gamla Stan – A special place in that it has the ability to ensure you wander around for hours without even realising it. This archaic island, with its tiny row houses, stuffed cafes and small squares is a must-see, even if you loathe touring historical sites (like me). Also, don’t forget to try out some ice cream. There are some mesmerising flavours available throughout the area. My favorite is ‘Stikki Nikki.’

Common Applications of Time Series Data (Python x Pandas)

Firstly, time in Python can be represented as either a string or a timedelta or datetime object. We’ll have more on those two below. Whenever reading data from a .csv file using the pd.read_csv function, time is generally read as a string, which can be converted to a datetime object with a very simply function. But more importantly, it is imperative to understand how time is actually represented.

YYYY-MM-DD HH:mm:ss.msmsms (m is minute, s is seconds, ms is milliseconds).

This notation is both simple and effective when dealing with time. Now, while reading, your code sees this format as a string, so it doesn’t whether this represents time or not. The pd.to_datetime is a vectorized function that converts all values in a Series (for dataframes, a specific column) to datetime objects. Intuitively, the first half is the ‘date’, and the second part the ‘time.’

Perhaps the most important element of dealing with time in python is your purpose. Generally, one would use this data to either calculate the amount of time passed between readings, or to calculate the amount of time between a starting time and current time. Let’s look at each separately.

Between Readings

To start off, I’ll write down an example. Say you have a pH sensor that gives you the following values: 7, 8, 9, 8. Now, your computer also logs the time at which these values are received: 2019-06-19 4:03:12, 2019-06-19 9:06:10, 2019-06-19 23:56:55 and 2019-06-20 1:01:04. Now, we have to find the amount of time passed between readings. Thus, you have a pandas dataframe as follows. I will explain why the last row is as such below.

        pH       time
0 7 2019-06-19 4:03:12
1 8 2019-06-19 9:06:10
2 9 2019-06-19 23:56:55
3 8 2019-06-20 1:01:04
Now, what we will do here is shift each time value down by 1 and store it in a series. This is done using the shift() function from pandas that takes in 1 required argument, n, that determines the number of spaces to shift by and whether to shift from left to right or top to bottom (default is top to bottom).
 
shifted_time = df.time.shift(1)

creating this series

0    NaN
1    2019-06-19 4:03:12
2    2019-06-19 9:06:15
3    2019-06-19 23:56:55
name: shifted_time

now, creating a last row of NaN and 0 respectively ensures that our last value is taken into consideration. The difference can now be calculated as follows.

diff = [df.datetime[i] - diff[i] for i in diff[]
diff[0] = 0    # as initial value has nothing to be compared against
df['difference'] = diff

creating the following data frame and giving us the amount of time between reading

    pH           time               difference
0 7 2019-06-19 4:03:12 ~
1 8 2019-06-19 9:06:15 0 days 05:03:03
2 9 2019-06-19 23:56:55 0 days 14:50:30
3 8 2019-06-20 1:01:04 0 days 01:05:09
This difference can be converted to seconds by specific functions, such as dt.total_seconds This just does this:

    pH           time               difference
0 7 2019-06-19 4:03:12 ~
1 8 2019-06-19 9:06:15 18183
2 9 2019-06-19 23:56:55 53430
3 8 2019-06-20 1:01:04 3909
difference is in seconds now. note: subtracting two datatime objects automatically creates a timedelta object, no need to do manually for latest version of pandas

Now, these sort of computations have many real world applications. For example, formula 1. If you’ve ever noticed, the person who comes first (Hamilton most of the time) has his time written in absolute terms, i.e, his time is written as it is. This is done by initializing the time column of the first row to 0. Then, from second place onwards, their time is represented as the difference by which they lost to the first person. I suggest you google F1 and have a look at it yourself.

Differences from starting point

This methodology isn’t so prominent to customers, but is extremely useful to data scientists looking to check whether the time data they receive has breaks or not. This is done by taking the minimum time value (normally the first one), and then subtracting it from each value – giving the time passed from the start to that specific point. These ‘differences’ are then plotted on the vertical axis, with the index of the time data in the entire data set taken on the horizontal axis.

Consequently, you get graphs like these that display the continuity of the data you have.

A log scale has been taken on the y-axis to ensure that small difference values are noticeable.

Now, Python, and especially pandas has many functions and methods to deal with time series data. This link would be the best to explore these functions –> https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html. One functionality that I absolutely adore is the ability to convert data as per timezone – adding both flexibility and adaptability to your code. This is extremely prominent on the web. For example, if you’ve ever noticed, most websites display the timings for a sport event as per your timezone. This is independent from python and pandas, but has the same functionality. Using your location, the time data employed by the server is converted to your timezone