PyCascades-2023

A huge welcome to everyone leading into our morning talks.

Ben and Eliza will welcome us to the first in-person PyCascades since 2020!

Did you know you can (mostly) replicate all of Python 3.8's syntax using only 11 core pieces of syntax (plus some magical functions)? Come to this talk to hear about how "simple" things like `+` and `.` actually unravel into chunks of Python code!

I spent over two years writing a blog post series about how various parts of Python's syntax actual unravel to chunks of code (https://snarky.ca/tag/syntactic-sugar/). In that time I learned how Python 3.8's syntax can be built up from just 11 key pieces of syntax and some functions provided by the runtime. In this talk I will go over those 11 syntactic constructs and why they can't be removed and go over various examples of everyday syntax like `+` and how it devolves into ordinary Python code (and thus how Python makes its syntax work).

Distributed tracing is a powerful tool for understanding microservices. Join this talk to learn what distributed tracing is, how to add it to your Python applications, an overview of how it works, and some reflections from someone who has implemented it across projects.

This talk aims to be a practical introduction to implementing distributed tracing. Rather than just introduce the concepts of tracing, people who listen to this talk should feel like they could get started tracing their own applications. We’ll show and provide frequent code samples to help people get started. There are many open source options for tracing Python applications, but we'll focus on opentelemetry for constructing traces, flask and FastAPI for our web servers, and Jaeger for visualizing traces.

Django being one of the most popular Python-based web frameworks, applications developed using Django are always on the radar of hackers who try to find the vulnerabilities in the Django application and exploit the same for their benefit. This talk will discuss how to secure Django Web applications.

Security is of utmost importance to most applications in general and web applications in particular. Django being one of the most popular Python-based web frameworks, applications developed using Django are always on the radar of hackers who try to find the vulnerabilities in the Django application and exploit the same for their benefit. Many times security is ignored or not well done due to a lack of awareness and the cost associated with it. But Security is too costly to be ignored. Although Django has many built-in security features, they are not sufficient to safeguard the application. The talk begins with highlighting the importance of security and identifying security issues in Django applications using the Mozilla Observatory tool, then using the recommendations of the tool to secure them. Next, I will compare and contrast Mozilla’s Web Security recommendations and Open Web Application Security Project(OWASP) Top 10 recommendations. Next, I will discuss built-in security features in Django. Finally, I will discuss the configuration settings and issues that may affect the secure deployment of Django applications. Outline 1. Importance of Security with respect to Web applications (02 Minutes) 2. Identifying security issues using Mozilla Observatory (05 Minutes) 3. OWASP Top 10 issues and how to address them in Django (08 Minutes) 4. Built-in Security features in Django (08 Minutes) 5. Secure Deployment of Django applications (04 Minutes) 6. Summary and Questions (03 Minutes)

What happens on the computer when you run print(“Hello world”)? This talk attempts to dissect how Python code gets translated for execution. While many programmers can live without interacting with compiler internals, a stronger understanding of CPython can help make us better programmers.

Python is a widely-used programming language thanks to its simple code syntax, but this convenience is brought by the hard work of the compiler. Among the many implementations (like PyPy and JPython), we will look at the most widely used CPython implementation. The talk will look at the compilation steps, and examine how the resulting bytecode will be executed in the Python virtual machine. Before a simple string is displayed on a terminal, CPython processes code performs complicated tasks from lexical analysis, parsing, compilation, and to evaluation loop. Even executing a simple program like print(“Hello world”) requires this laborious process performed instantly on the computer. By understanding the execution pipeline, we can identify execution bottlenecks to improve code performance.

One of the things that makes Python easy to work with is that you don't need a compile step before you can check that your code works. What if you could skip the save and run steps, too? Live coding lets you constantly run your code, as you edit it, and shows you what's happening inside. It's also great for visual tools like Matplotlib and Pillow. I'll be demonstrating my Live Coding in Python project that includes plugins for PyCharm, Emacs, and SublimeText, as well as live tutorials that use Pyodide to let the reader experiment with code samples right in the browser.

* 5 mins - Motivation and inspirations * Faster feedback loops give us more control. We can experiment and explore. * Bret Victor talks, like Inventing on Principle * Visualising how our code runs makes it easier to understand and change. * Seymour Papert's turtle graphics * 10 mins - Turtle graphics demo * Playing with turtle graphics * Exploring the fill algorithm to find unexpected patterns * Matplotlib graphics, too * Pillow graphics, as well * 5 mins - Code visualisation demo * Useful for real code * Driven by unit tests * 4 mins - Future work * Browser tutorials * Visual unit tests * 1 min - Contribution opportunities * Writing tutorials - it's just markdown! * More plugins - VS Code and others * New Python versions

As software is increasingly integrated with many third party components, particularly open-source components, it is essential to have a clear understanding of all of the software that is being deployed. With an increasing focus on improving the Cybersecurity of the many different parts of the supply chain, there is a growing expectation that a Software Bill of Materials (SBOM) will become a key artefact of any software component to help capture all of the software assets being used. This talk will briefly introduce the concept of SBOMs and show how a number of Python tools will help in the production, management and use of SBOMs as part of a system lifecycle.

The US Executive Order 14028 (improving the nation's cybersecurity) released in May 2021 significantly raised the profile of Software Bill of Materials (SBOMs) as an important artefact to help in improving the cybersecurity of deployed products and systems. However despite a growing awareness of SBOMs in the past 2 years, particularly in some market sectors, the adoption and use of SBOMs is still not widespread. In part this is due to the SBOM use cases not being understood and in part due to the lack of suitable tooling to help in the generation (although some signficant progress is now being made), the management and use (or consumption) of SBOMs as part of ongong system processes. This talk will identify some key use cases which SBOMs can be used and then identify how a number of open source tools and libraries (all developed in Python) can help. There will be a particular emphasis on showing how the tools can use SBOMs to help understand the risks associated with the use of a software artefact, particularly those produced (and hopefully maintained) by others.

"But it worked on machine" is one the most frustrating lines to hear when collaborating on a project. Creating and configuring reproducible environments is a major part of modern software development and had led to the popularity of tools like Docker to specify where and how code runs. Setting up Docker and Development Containers in VS Code make it easy to configure not only the where the code runs, but also the developer workspace. Setting up these tools can reduce effort for maintainers, bootstrap contributors, and make running events like workshops or sprints easier. In this talk, we will cover why setting up container infrastructure like Docker can be useful for your project, and how you can extend that with Dev Containers to configure a full development experience in VS Code. We will also take a look at two common OSS project scenarios and how workflows for using Docker, Dev Containers and Codespaces make things easier. No container experience required, and some knowledge of VS Code helpful, but not necessary.

Outline for the talk: - 3 min | Intro: "Obedient slumbers that can wake and weep" - Why can contributing/collaborating on OSS projects be technically hard? - How container platforms like Docker have helped improve reproducibility - 7 min | Dev Containers: "Desires composed, affections ever even" - Docker setup is portable, but not necessarily editor specific - Specifying an entire VS Code dev experience: JSON with optional Docker file - Quick look at the .decontainer file spec - Sample repo and demo starting experience when cloning a new repo - 7 min | Contribution + Collaboration: "Each PR accepted, and each issue closed" - Can help simplify/exemplify CONTRIBUTING.md - Easily share how you develop with contributors or create a container that demos a problem to share - Future: Dev Containers can be shared with CI like GitHub actions, so it is using the exact same env as well - 7 min | Workshops: "Unequal task! a passion to resign" - Dev containers create accessible and free, out-of-the-box experiences for teaching spaces - Users can launch and use entirely from the browser - Different levels of hardware backend compute available - 1 min | Review "Labour and rest, that equal periods keep"

Metaprogramming is writing programs that manipulate programs. Dive into the deeper python magic of metaprogramming and enhance your understanding of object orientation in Python and its working under the hood.

> "Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why)." -Tim Peters Metaprogramming is writing programs that manipulate programs. Dive into the deeper python magic of metaprogramming and enhance your understanding of object orientation in Python and its working under the hood. Metaclasses are used in the internal implementation of Python enums and data classes, and are used in designing various libraries including web servers (Django) and ORMs (Mongoengine). Get a head start into the metaprogramming, deepen your understanding of pythonic object lifecycle and see the power of metaclasses in action.

When it comes to choosing a light, simple and high performance database for your python application, SQLite surely comes to mind. Learning and befriending SQlite has several benefits, and it is a must have tool for every programmer. But have you ever wondered what is happening behind the scenes which makes its storage and retrieval so efficient? How is data being managed internally? Let's explore the SQLite database using python while learning some intriguing underlying principles.

As applied programmers, we are often not concerned with low-level realizations; however, in order to make optimal decisions and select the best database for our application, it is critical to understand how data is being handled under the hood. In this session, we will use python to explore SQLite and build a simple application. Through this session, you will get familiar with the internal workings of the SQLite database. You will learn how SQLite uses b-tree indexing for efficient searches and how data is actually stored on disk. We will also observe how the querying capabilities of SQLite are as powerful as any other SQL database. Along the way, you will understand some of the advantages and trade-offs of using SQLite and when it is the best fit. This would also serve as a stepping stone in making better design decisions and effectively conveying them, even if you lack prior experience in the problem domain.

The Zen of Python, the well known list of 20 Python aphorisms (of which only 19 are written down), contains a lot of prescriptive advice for what makes "Good" Python code… or it would, if one of them didn't give you permission to ignore the rest of them. Focusing on one piece of the 19 aphorisms, this talk explores the contradictions in the Zen of Python's advice, and how even with prescriptive rules, writing "Good" Python remains a subjective exercise.

When you type “import this” into a Python interpreter, you get PEP-20, 20* aphorisms that describe what “Good” Python code looks like. Most of these are prescriptive, and have been used to settle countless arguments about how Python should grow, or how new ideas should be implemented. But in these 20 aphorisms (which you may know as the “Zen of Python”), there are some inherent contradictions. When “explicit is better than implicit” and “simple is better than complex”, what does it mean when being explicit means you must introduce complexity? When “readability counts” and “special cases aren’t special enough to break the rules”, what does it mean when making code more readable presents a special case in your codebase? The answer lies in one often-forgotten aphorism: “practicality beats purity”. When you can weigh practicality as a factor, you can decide that other factors are restrictive, and discard them as “impractical”. This idea shows up in many places in Python: Type hints can help you to be explicit, but they introduce complexity. Focusing on practicality means you can adopt type hints only when you need them. Decorators are simple, but can be used to introduce ambiguity. In many cases, the practical benefits of decorators outweigh their potential for misuse. But how far can the pursuit of practicality let you go? Does practicality let you treat the other 19 aphorisms as optional? Does practicality give you an escape hatch from the rest of the Zen of Python? In this talk, we’ll look at how practicality has been a guiding principle in the evolution of Python. We’ll look at how to weigh practicality in the face of the other 19 aphorisms of PEP-20, and we’ll look at how focusing on practicality can guide you towards these other attributes of good Python. --- (* per PEP-20, the Zen of Python contains “20 aphorisms, only 19 of which have been written down.”)

Wrapping up our first day of talks.

Ben and Eliza will wrap up the first day of PyCascades and share notes for the next day.

Welcome to PyCascades 2023 day 2.

A few reminders and notes for the second day of talks.

Digital sensors have become a standard way of collecting data in the physical world. Join me as we delve into the realm of digital bus sensors by creating a driver for an I2C light-to-digital converter.

Digital buses are used today in most electronic devices to connect components to a microcontroller or PC. This talk will present a workflow for writing Python I2C bus drivers using classes and JSON. We'll tackle how to read a manufacturer's datasheet, read and write I2C registers and decode byte data. We'll also look at how to source other interesting sensors and interface with them.

Infrastructure From Code (IfC) is a relatively new trend in Cloud DevOps automation aiming at the automatic generation of cloud deployment templates directly from the application code. There are already several solutions primarily focused on the TypeScript programming language. This talk is about what could be done for the Python ecosystem.

Infrastructure as Code - the use of repeatable scripts or SDKs to configure the cloud environment for an application - has expanded the possibilities of Serverless with Continuous Delivery. However, there are fundamental issues with IaC: it is a time-consuming and tedious process to prepare the scripts, the activity requires the involvement of DevSecOps experts, and the scripts are only loosely related to the application code. For any non-trivial application, these scripts grow very quickly in complexity and size. Applying IaC to optimize the configuration of the test, staging, and production environments only exacerbates these issues. Infrastructure from Code is a recent advancement over IaC that addresses these obstacles. IfC interprets a mainstream programming language code and automatically generates the specifications needed to configure a cloud environment. So far, the most advanced solutions, such as [ServerlessCloud](https://www.serverless.com/cloud), [Ampt](https://getampt.com/), or [Nitric](https://nitric.io/), were proposed for the TypeScript ecosystem. While, for Python, [AWS Chalice](https://aws.github.io/chalice/index) is a cloud-vendor-specific solution and offers a relatively limited feature set. In this talk, I'm going to clarify the IaC paradigm and its limitations, analyze the exact problems that IfC is supposed to solve, compare recent IfC offerings, and propose what could and, in my opinion, should be done for the Python ecosystem. I will also share my personal experience of implementing such a system, code-named [Cloud AI Operating System (CAIOS)](https://www.caios.io/home): what went well, what still needs to be resolved, and where, in my view, the Python community could contribute in making Python as the number one choice for cloud programming language.

pytest rocks, obviously. When people start using pytest as a team, they often come up with cool fixtures that would be great to share across projects. In fact, many great Python packages come pre-loaded with pytest fixtures. This talk describes how easy it is to share fixtures using the pytest plugin model.

There are a lot of people creating awesome pytest fixtures to help them with their testing. In order to share those fixtures with others, we need to discuss Python packaging and pytest plugins. Both of those topics, building plugins and packaging, have a bad rap due to their complexity in the past. But things have gotten a ton better with packaging. And when you just want to share a fixture, you don't really need to know much about the pytest plugin system. This talk walks through a small fixture example, then walks through how to make it into a plugin, and how to package it. It's a very gentle onramp to modern packaging. The example will discuss show code exampes for setuptools, flit, and hatch, all using pyproject.toml

After learning doing in Python, we started multiple Python or Data Science projects. Dependency management becomes a skill that we need to avoid requirement conflicts amount projects. In this talk, we will learn how dependencies management tools work and how to choose the right one to use.

In this talk, we will use venv, the environment managing tool that come with CPython, and conda, the tool that is popular among data science partitioners to decipher how environment and dependency management works. First, we will go through some basic knowledge of how Python "sees" your package and know where to import them when needed. Then, we will make the audience aware that, to avoid the requirement conflict amount different projects, we need a new set of environments for each project. Then, we will see how tools like venv and conda can create a new set of environments. We will cover both the practical aspect - how to use the tools and which one to use, and the theoretical aspect - the inner working of the tools and why they are different. This will make the audience get familiar with some tools that are at their disposal and inspire them to start using them for their work. This talk is for beginner Pythonistas or data scientists who started to use Python professionally and those who are curious to know how the environment and dependency management tools work. ## Outline - Introduction (5 mins) - How Python sees your packages (5 mins) - Why we need separate environments (1 mins) - How venv create new environment (2 mins) - How conda create new environment (2 mins) - Comparing venv and conda - which one to use (5 mins) - Conclusion (5 mins)

Secure software supply chains with 0 vulnerabilities sounds like a great idea, but once you start looking through entire dependency chains and large systems, it can be a lot harder to achieve than one might expect. Using the free, open source, CVE Binary Tool vulnerability scanner (written in python!), we'll show what it looks like to set up vulnerability scanning, what kinds of fun things you find, and how keeping things up to date can mean an ongoing maintenance burden that is more like a free puppy than a free beer. We'll talk about how naive policies, governmental mandates and capitalism may ruin your day, and what we can do to stay secure and help everyone get past the puppy phase without sending anyone back to the pound.

Why should you care about known vulnerability scanning? (3 minutes) - What is supply chain security? - Why are governments and major companies freaking out about it right now? Why should you listen to me? (2 minutes) - brief intro/credentials/experience: I've got a phd in security, work in open source security for a major company, and maintain a tool for open source teams to do vulnerability scanning for free - also, a picture of my puppy and explanation of how I understand "free as in puppies" a lot more deeply now How do I do a known vulnerability scan? (5 minutes) - This will focus on using CVE Binary Tool (because it's what I know best and it should be interesting to a pyCascades audience) but I will mention some other tools available for free to open source developers - CVE Binary Tool is free, open source and written in python. Lots of our new features have been developed as part of the Python Software Foundation's summer mentoring program under the Google Summer of Code banner. - Two types of scans: 1. scans to find components -- how does that work? 2. scanning a list of known components -- how to prepare a software bill of materials (SBOM) What do I do when it finds something? (5 minutes) - how to triage a scan & fix issues - Types of unfixed issues, how they happen, and what to do: 1. Fix is still in progress 2. Issue is disputed/incorrect 3. Issue is reporting an intentional behaviour 4. Developers will not fix the issue Why is known vulnerability scanning going to ruin the world? (8 minutes) - Is it more expensive to fix vulnerabilities or prevent people from reporting them? - Can you ever have a large system with 0 known vulnerabilities? - What should we be doing about the quality of publicly known vulnerabilities, if anything? - Is it possible to make policies that won't ruin everything? Conclusions (2 minutes) - How puppy raising & security are great (except when they aren't) - how both get better after a few months and some experience!

Code coverage is a great metric - how much of your code is actually tested by your unit tests. However, this doesn’t tell you how good your tests actually are at picking up changes to your codebase - if your tests aren’t well thought-out, it’s possible for changes to get past your unit tests but break production. Mutation testing is a great (and massively underrated) way to understand how valuable your tests are. Mutation score quantifies how much trust you should put in your tests to stop mutant versions of your code escaping from your lab and wreaking havoc in production. In this talk, I’ll show you examples of how to get started with mutation testing and how to integrate it into your CI/CD pipeline. After the session, you’ll be ready to use mutation testing with wild abandon and you'll be ready to integrate it into your release engineering process, testing your mutant code without needing to set foot in the lab at all!

Developers often use metrics as targets. Code coverage is one of the most common, and most prone to corner-cutting and abuse. Goodhart's Law states: When a metric becomes a target, it ceases to be a good metric (and trust me, I have a lot of memes to share with you about this!) Mutation score is different. Because it tests your tests, coverage isn't enough - you have to write tests that are resistant to all the mutated code that wants nothing more than to sneak through your CI/CD pipeline into production.

Parsing tools aren't just for compilers! You can use parsing tools to read, parse, and manipulate all sorts of data. We'll use Lark, a Python parsing library, to look at some basic examples and the resulting parse trees, demonstrating the end-to-end process of input, parse tree, and output. This talk is for anyone that has used Python to read in data from a text file.

People generally associate parsers with writing compilers. While few people write compilers, many people write code to read, interpret, and manipulate data. Regular expressions are a popular choice. For more complex data, a parsing tool could be the better choice, because the grammar is more readable than a regular expression, and the parse tree is an abstract data type that reflects the nature of the data. In this talk, we'll use Lark, a Python parsing library, to learn what a grammar looks like, what a parse tree is, and how to use the parsing process to create a new output from an input. A brief outline: The talk walks through a very basic parsing example, add-numbers, that parses an equation like "2 + 3" and returns the solution, "5". Topic | Time | Description ---|---|--- What is parsing? | 5 minutes | The basic task of parsing is to read the input, create the parse tree, and then output the data in a new form. What is a typical input for parsing? For add-numbers, what will the input look like, and what output do we want? What is a grammar? | 5 minutes | What is a grammar and why do we need one? What does a grammar look like? What is and isn't supported? What is the add-numbers grammar? What is a parse tree? | 5 minutes | How does the structure of the parse tree mimic the structure of the data? Related concepts include listeners and visitors. What is the parse tree for add-numbers? What is the output? | 5 minutes | How do we use the parse tree to do create the output? Parsing in practice | 5 minutes | How can we extend the add-numbers example? Other than compilers, what are some use cases for parsers?

Python has a variety of looping mechanisms for iterables engineered using functions and language keywords. Is it possible that two loops with same complexity have different runtimes? Which mechanism is the fastest and the slowest, Which one should you use to optimize your application?

Python has a variety of looping mechanisms for lists built using functions and language keywords. But each one of them has its pros and cons. A few mechanisms that we’ll cover are: - For Loop - Ranged Loops - Enumerate method - While Loop - List Comprehensions - Generators Expressions - Zip method In this session, we’ll do a deep analysis on various different looping mechanisms, how they operate, and what time and space complexities they carry. We’ll also talk about a run-time of various algorithms and how they perform using these mechanisms. We’ll also go through the comparison between the usage and identify how we can improve the performance of our algorithms by use of different looping mechanisms. **Audience** This tutorial is for Python programmers who are familiar with looping in Python but haven’t fully embraced the complexities of looping mechanisms in Python and cannot comprehend the ideal usage in various circumstances. **Outline:** - Introduction to loops in Python - Describe each loop with an example - What’s happening inside each loop - Which functions are called? What objects are returned? - How memory is managed? - Performance analysis on various different algorithms. - What loop to use in what algorithm - Analysing the target of an algorithm - Categorizing the targets according to each loop - Analyzing the final outcome

With a little wiring and a few lines of code, one can create your own hyperlocal air sensor system to send alerts, activate smart-windows, and gather data. A sensor measures air particulate density and sends the values to a Raspberry Pi compute unit. The Raspberry Pi then sends the data along with a timestamp and location information to a cloud instance of Redis, a NoSQL data store. Once in Redis, the data can trigger SMS notifications, feed data visualization libraries, or activate electronics such as air purifiers or motors to open or close windows. This can be scaled to monitor air quality in multiple rooms, offices, city blocks, or cities.

This project was created when air quality was at an all-time low on the west coast of North America due to wildfires. Using an inexpensive and widely available air sensor module and a Raspberry Pi computer, I created a small portable device that would serve as a perfect platform to deliver air quality data points to a database in the cloud. The Raspberry Pi accepts incoming sensor data, converts it to a timestamped entry in a timeseries, and sends the payload to a Redis instance in the cloud. The data is then able to feed visualization software (finale of presentation), send alerts via email/SMS, or activate servo motors to open or close windows or activate air purifiers. All code is written in Python. Communication with Redis is done with redis-py, a client library providing Redis commands. Talk attendees should have a comfortable grasp of Python, basic data structures, and a novice understanding of NoSQL databases.

Final words for the main conference days!

Ben and Eliza will wrap up our last day of the main talks for PyCascades 2023, and speak about sprints happening the next day.