How Beanstalk handles multiple workers

Beanstalk is a neat little job queue. I stumbled across it years ago and wanted to know how it worked, so I wrote a little C# client library called Turbocharged.Beanstalk to teach myself. I also learned a lot about socket programming.

The hardest concept for me to understand was how Beanstalk handles a worker that works on multiple jobs on the same TCP connection. At first, it looks like Beanstalk doesn’t support this. NSQ has a max_in_flight setting, but Beanstalk has no equivalent.

Beanstalk’s DEADLINE_SOON message is what allows clients to reserve multiple jobs on a connection without breaking.

Someone asked me over email whether Beanstalk supports multiple workers, so I thought I’d write it down and put in public in case it helps anyone else.

Summary

Beanstalk supports reserving multiple jobs on a single TCP connection. Because a pending reserve blocks the connection until a job arrives, the Beanstalk server will abort any in-flight reserve commands with DEADLINE_SOON if a reserved job is about to expire but hasn’t been deleted yet.

This unblocks the connection and Beanstalk can process any delete commands that were waiting to be processed.

Definitions

producer - A client connected to Beanstalk that puts jobs in a tube.
worker - A client connected to Beanstalk that pulls jobs out of a tube and processes them.
job - A job is just a sequence of bytes. It might be binary data, or it might be text or JSON in UTF-8. Beanstalk doesn’t care, it just sees bytes.
tube - A named container where jobs are stored in Beanstalk. Producers put jobs in a tube and workers watch tubes that contain jobs they can process.

Workers

A client connects to Beanstalk and sends commands.

Beanstalk responds to a client’s commands in order. A client can send multiple commands at once, but Beanstalk will not respond to subsequent commands until it has responded to previous ones.

When a client connects and wants to be a worker, it issues these two commands:

CLIENT COMMAND              SERVER RESPONSE
-------------------------------------------------
watch tweets
                            WATCHING 1
reserve

This begins watching the tube named “tweets”. This connection is now blocked until the server can respond to the reserve command.

Eventually, there will be a job ready in the tube. Beanstalk responds to the reserve command with this:

CLIENT COMMAND              SERVER RESPONSE
-------------------------------------------------
reserve
                            RESERVED 1 21
                            this is my cool tweet

This job has ID #1 and is 21 bytes long. The job itself is “this is my cool tweet”.

When the client processes this job (by posting it to Twitter), it sends the following command:

CLIENT COMMAND              SERVER RESPONSE
-------------------------------------------------
delete 1
                            DELETED 1

This deletes job ID #1. (The client could instead bury or release the job, but those commands aren’t important for this topic.)

Now the connection is idle, so the worker will probably send another reserve command. The worker issues these reserve/delete commands forever to process jobs as they come in.

Reserving multiple jobs

If a worker can process multiple jobs concurrently, it can send the reserve command multiple times.

CLIENT COMMAND              SERVER RESPONSE
-------------------------------------------------
reserve
reserve
                            RESERVED 1 21
                            this is my cool tweet

The client has reserved one job, but there are no other jobs in the tube, so the second reserve command has not been answered yet.

The client finishes processing job #1 and sends a delete command:

CLIENT COMMAND              SERVER RESPONSE
-------------------------------------------------
delete 1

But the server won’t process this delete because it hasn’t responded to the previous reserve command yet (because the tube is empty).

This looks like a bad situation: The server won’t process the delete until it responds to the reserve, and it won’t do that until a new job is put in the tube.

But maybe a new job will never be put in the tube. How does Beanstalk ensure job #1 gets deleted?

Deadlines

If a worker gets stuck processing something, you don’t want that job to be lost forever. So all jobs have a time to run. If a worker reserves a job and doesn’t handle it within the time to run, Beanstalk assumes the worker has failed and redelivers the job to another worker.

If a worker has a reserved a job that reaches 1 second left in its time to run, any pending reserve commands will abort with the message DEADLINE_SOON.. This is the magic that fixes the previous problem.

The complete exchange looks like this, assuming a 30 second time to run:

CLIENT COMMAND              SERVER RESPONSE
-------------------------------------------------
reserve
reserve
                            RESERVED 1 21
                            this is my cool tweet
delete 1

         ...29 seconds go by...

                            DEADLINE_SOON
                            DELETED 1

The DEADLINE_SOON is the response to the second reserve command. This unblocks the connection, and now Beanstalk immediately processes the delete command.

DEADLINE_SOON doesn’t indicate any sort of failure, it’s just there to unblock the connection so any pending delete commands can go through.

Posted Sunday, February 18, 2018

Converting Turbocharged.Beanstalk to .NET Standard (Part 3)

This is part 3 of a series about converting Turbocharged.Beanstalk to .NET Standard so it can run on .NET Framework, .NET Core, and Mono.

Part 1: Converting the project files and getting the main project to build
Part 2: Getting the test project to build
Part 3 (this post): Fixing the AppVeyor build

In part 2, I had just created this commit.

The tests are passing, but the AppVeyor build is failing.

Fixing the AppVeyor build

The output from the build ends with this:

msbuild "C:\projects\turbocharged-beanstalk\src\Turbocharged.Beanstalk.sln" /verbosity:minimal /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll"
Microsoft (R) Build Engine version 14.0.25420.1
Copyright (C) Microsoft Corporation. All rights reserved.
C:\projects\turbocharged-beanstalk\src\Turbocharged.Beanstalk\Turbocharged.Beanstalk.csproj(1,1): error MSB4041: The default XML namespace of the project must be the MSBuild XML namespace. If the project is authored in the MSBuild 2003 format, please add xmlns="http://schemas.microsoft.com/developer/msbuild/2003" to the <Project> element. If the project has been authored in the old 1.0 or 1.2 format, please convert it to MSBuild 2003 format.
C:\projects\turbocharged-beanstalk\src\Turbocharged.Beanstalk.Tests\Turbocharged.Beanstalk.Tests.csproj(1,1): error MSB4041: The default XML namespace of the project must be the MSBuild XML namespace. If the project is authored in the MSBuild 2003 format, please add xmlns="http://schemas.microsoft.com/developer/msbuild/2003" to the <Project> element. If the project has been authored in the old 1.0 or 1.2 format, please convert it to MSBuild 2003 format.
Command exited with code 1

This AppVeyor image is using MSBuild 14, which cannot read the new project file format. I need to use an image supporting the new format.

I could do this in the AppVeyor settings, but then these settings wouldn’t follow any forks of the repository. Instead, I’ll create an appveyor.yml in the root of the repository.

version: 1.0.{build}
image: Visual Studio 2017
configuration: Release

before_build:
  - cmd: dotnet restore

build:
  - verbosity: minimal

test_script:
  - cmd: dotnet test src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj

after_build:
  - cmd: dotnet pack src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj --configuration Release

artifacts:
  - path: '**/*.nupkg'

And now commit and push.

Build started
git clone -q --branch=netstandard https://github.com/jennings/Turbocharged.Beanstalk.git C:\projects\turbocharged-beanstalk
git checkout -qf 66f9673efad5dbdd31453382787570fc697e43d8
dotnet restore
MSBUILD : error MSB1003: Specify a project or solution file. The current working directory does not contain a project or solution file.
Command exited with code 1

Oh poop. The build script expected the solution file to be in the root of the repository.

I could add cd to all the build scripts, or I could just move the solution file.

Guess which one is easier.

❯ git commit -m 'move the solution file to the root of the repository'
[netstandard c45da68] move the solution file to the root of the repository
 1 file changed, 3 insertions(+), 3 deletions(-)
 rename src/Turbocharged.Beanstalk.sln => Turbocharged.Beanstalk.sln (91%)

❯ git push

And now AppVeyor says:

Build success

Yay!

Posted Saturday, June 17, 2017

Converting Turbocharged.Beanstalk to .NET Standard (Part 2)

This is part 2 of a series about converting Turbocharged.Beanstalk to .NET Standard so it can run on .NET Framework, .NET Core, and Mono.

Part 1: Converting the project files and getting the main project to build
Part 2 (this post): Getting the test project to build
Part 3: Fixing the AppVeyor build

At the end of part 1, I had just created this commit. Time to try to get the unit tests working.

Getting the test project to build

First I’ll just try building and see what happens.

src/Turbocharged.Beanstalk❯ cd ../Turbocharged.Beanstalk.Tests

...urbocharged.Beanstalk.Tests❯ dotnet restore; dotnet build

	...a whole lot of output...

...
FakesAndMocks/FakeConsumer.cs(15,41): error CS0246: The type or namespace name 'Job' could not be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
SerializationFacts.cs(21,9): error CS0246: The type or namespace name 'ConnectionConfiguration' could not be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
SerializationFacts.cs(22,9): error CS0246: The type or namespace name 'WorkerOptions' could not be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(477,21): error CS0103: The name 'WorkerFailureBehavior' does not exist in the current context [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(478,21): error CS0103: The name 'WorkerFailureBehavior' does not exist in the current context [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(479,21): error CS0103: The name 'WorkerFailureBehavior' does not exist in the current context [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(480,21): error CS0103: The name 'WorkerFailureBehavior' does not exist in the current context [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(481,81): error CS0246: The type or namespace name 'WorkerFailureBehavior' could not be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(544,21): error CS0103: The name 'WorkerFailureBehavior' does not exist in the current context [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(545,21): error CS0103: The name 'WorkerFailureBehavior' does not exist in the current context [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(546,21): error CS0103: The name 'WorkerFailureBehavior' does not exist in the current context [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(547,21): error CS0103: The name 'WorkerFailureBehavior' does not exist in the current context [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(548,86): error CS0246: The type or namespace name 'WorkerFailureBehavior' could not be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(16,9): error CS0246: The type or namespace name 'IConsumer' could not be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(17,9): error CS0246: The type or namespace name 'IProducer' could not be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
    0 Warning(s)
    38 Error(s)

Time Elapsed 00:00:02.27

38 errors? Ugh.

Wait… most of these errors are from not being able to resolve anything from Turbocharged.Beanstalk. Oops. That’s because I didn’t add it as a project reference.

...urbocharged.Beanstalk.Tests❯ dotnet add reference ..\Turbocharged.Beanstalk\Turbocharged.Beanstalk.csproj
Reference `..\Turbocharged.Beanstalk\Turbocharged.Beanstalk.csproj` added to the project.

And I’ll try building again.

...urbocharged.Beanstalk.Tests❯ dotnet build
Microsoft (R) Build Engine version 15.1.1012.6693
Copyright (C) Microsoft Corporation. All rights reserved.

    ...output...

/usr/local/share/dotnet/sdk/1.0.3/Microsoft.Common.CurrentVersion.targets(1964,5): warning MSB3277: Found conflicts between different versions of the same dependent assembly that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
obj/Debug/netcoreapp1.1/Turbocharged.Beanstalk.Tests.AssemblyInfo.cs(6,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
obj/Debug/netcoreapp1.1/Turbocharged.Beanstalk.Tests.AssemblyInfo.cs(7,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
obj/Debug/netcoreapp1.1/Turbocharged.Beanstalk.Tests.AssemblyInfo.cs(8,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
obj/Debug/netcoreapp1.1/Turbocharged.Beanstalk.Tests.AssemblyInfo.cs(9,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
obj/Debug/netcoreapp1.1/Turbocharged.Beanstalk.Tests.AssemblyInfo.cs(11,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
obj/Debug/netcoreapp1.1/Turbocharged.Beanstalk.Tests.AssemblyInfo.cs(12,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
obj/Debug/netcoreapp1.1/Turbocharged.Beanstalk.Tests.AssemblyInfo.cs(13,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
ConnectionFacts.cs(3,14): error CS0234: The type or namespace name 'Configuration' does not exist in the namespace 'System' (are you missing an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
SerializationFacts.cs(3,14): error CS0234: The type or namespace name 'Configuration' does not exist in the namespace 'System' (are you missing an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
    1 Warning(s)
    9 Error(s)

Time Elapsed 00:00:02.35

Okay, I’ve got the same AssemblyInfo.cs problem from last time. Delete that file and I’ve fixed seven errors.

The only two errors left are because System.Configuration isn’t available. Instead, I’ll use Microsoft.Extensions.Configuration to read configuration out of a JSON file.

...urbocharged.Beanstalk.Tests❯ dotnet add package Microsoft.Extensions.Configuration
...urbocharged.Beanstalk.Tests❯ dotnet add package Microsoft.Extensions.Configuration.Json
...urbocharged.Beanstalk.Tests❯ dotnet add package Microsoft.Extensions.Configuration.FileExtensions
...urbocharged.Beanstalk.Tests❯ dotnet restore

Now I’ll replace all references to ConfigurationManager with IConfigurationRoot.

And… go!

...urbocharged.Beanstalk.Tests❯ dotnet build

	...output...

Build FAILED.

MiscellaneousFacts.cs(16,48): error CS1061: 'Type' does not contain a definition for 'Assembly' and no extension method 'Assembly' accepting a first argument of type 'Type' could be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:02.62

What? Oh, I’m using Phil Haack’s trickery to avoid accidentally creating async void tests. It uses reflection, and apparently Type no longer has a property named Assembly on it. I just need to change type.Assembly to type.GetTypeInfo().Assembly, and I’m good.

Okay, now to see if it builds:

...urbocharged.Beanstalk.Tests❯ dotnet build
Microsoft (R) Build Engine version 15.1.1012.6693
Copyright (C) Microsoft Corporation. All rights reserved.

  Turbocharged.Beanstalk -> /Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/bin/Debug/netstandard1.4/Turbocharged.Beanstalk.dll
  Turbocharged.Beanstalk.Tests -> /Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/bin/Debug/netcoreapp1.1/Turbocharged.Beanstalk.Tests.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.91

Hooray!

Verifying the tests pass

These tests require a running instance of Beanstalk to test against. I’ll run it locally using Docker.

...urbocharged.Beanstalk.Tests❯ docker create --name beanstalkd -p 11300:11300 schickling/beanstalkd
56d4329fc273977df85cb37a2837247a360b18dca3b1147d687eb7bcf7f8e908

...urbocharged.Beanstalk.Tests❯ docker start beanstalkd
beanstalkd

Okay, now I have Beanstalk running on port 11300. I’ll run the tests.

...urbocharged.Beanstalk.Tests❯ dotnet test
Build started, please wait...
Build completed.

Test run for /Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/bin/Debug/netcoreapp1.1/Turbocharged.Beanstalk.Tests.dll(.NETCoreApp,Version=v1.1)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:00.5299848]   Discovering: Turbocharged.Beanstalk.Tests
[xUnit.net 00:00:00.6858187]   Discovered:  Turbocharged.Beanstalk.Tests
[xUnit.net 00:00:00.7478363]   Starting:    Turbocharged.Beanstalk.Tests
[xUnit.net 00:00:00.8914508]     Turbocharged.Beanstalk.Tests.SerializationFacts.CanSerializeAndDeserializeAJob [FAIL]
[xUnit.net 00:00:00.8925155]       System.TypeInitializationException : The type initializer for 'Turbocharged.Beanstalk.Tests.Settings' threw an exception.
[xUnit.net 00:00:00.8925976]       ---- System.IO.FileNotFoundException : The configuration file 'settings.json' was not found and is not optional. The physical path is '/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/bin/Debug/netcoreapp1.1/settings.json'.

	...several dozen copies of this error...

Oh, the tests can’t find settings.json because it isn’t getting copied to the output directory. I need to add the following to Turbocharged.Beanstalk.Tests.csproj:

<ItemGroup>
    <None Include="settings.json" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>

And now:

...urbocharged.Beanstalk.Tests❯ dotnet test
Build started, please wait...
Build completed.

Test run for /Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk.Tests/bin/Debug/netcoreapp1.1/Turbocharged.Beanstalk.Tests.dll(.NETCoreApp,Version=v1.1)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:00.5141871]   Discovering: Turbocharged.Beanstalk.Tests
[xUnit.net 00:00:00.6738833]   Discovered:  Turbocharged.Beanstalk.Tests
[xUnit.net 00:00:00.7304119]   Starting:    Turbocharged.Beanstalk.Tests
[xUnit.net 00:00:06.1207816]     Turbocharged.Beanstalk.Tests.ConnectionFacts.ConnectWorker_FollowsFailureOptionsIfTheDeserializerThrows [SKIP]
[xUnit.net 00:00:06.1209331]       This test occasionally fails, especially in AppVeyor. I think it's due to timing of the worker getting shut down.
Skipped  Turbocharged.Beanstalk.Tests.ConnectionFacts.ConnectWorker_FollowsFailureOptionsIfTheDeserializerThrows
[xUnit.net 00:00:06.3554792]   Finished:    Turbocharged.Beanstalk.Tests

Total tests: 45. Passed: 44. Failed: 0. Skipped: 1.
Test Run Successful.
Test execution time: 6.9826 Seconds

Yay, it worked! I’d better commit before something breaks it.

...urbocharged.Beanstalk.Tests❯ git add -A

...urbocharged.Beanstalk.Tests❯ git commit -m 'make unit tests buildable and passing on .NET Core'
[netstandard 3e6d8b6] make unit tests buildable and passing on .NET Core
 7 files changed, 54 insertions(+), 44 deletions(-)

Posted Saturday, June 17, 2017

Converting Turbocharged.Beanstalk to .NET Standard (Part 1)

.NET Standard is the future of cross-platform .NET. By targeting .NET Standard, a library will run on any .NET implementation that supports it:

.NET Framework (Windows only)
.NET Core (cross-platform)
Mono (cross-platform)

But I find the ecosystem very confusing right now. The story used to be “just install .NET Framework version X, and target it in Visual Studio”, but now it feels like the tools and file formats are changing every few months (it isn’t, but it feels that way).

I’m going to finally figure out where the ecosystem is at by converting some of my projects to .NET Standard.

This series documents the process I followed to convert Turbocharged.Beanstalk to .NET Standard.

Part 1 (this post): Converting the project files and getting the main project to build
Part 2: Getting the test project to build
Part 3: Fixing the AppVeyor build

Installing the tooling

I’m doing this work on a Mac.

I already the tooling installed, but I think the only thing that needs to be installed is the .NET Core SDK. Here’s the output of dotnet --info on my system.

❯ dotnet --info
.NET Command Line Tools (1.0.3)

Product Information:
 Version:            1.0.3
 Commit SHA-1 hash:  37224c9917

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.12
 OS Platform: Darwin
 RID:         osx.10.12-x64
 Base Path:   /usr/local/share/dotnet/sdk/1.0.3

Creating the new solution and project files

First, a new branch to work in (I’m starting from this commit):

❯ git clone git@github.com:jennings/Turbocharged.Beanstalk

❯ cd Turbocharged.Benstalk

❯ git checkout -b netstandard
Switched to a new branch 'netstandard'

❯ git rev-parse --short HEAD
d5eb5d0

This project currently only works on .NET Framework and Visual Studio. Since I’m working from my Mac, the first step seems to be to convert the solution and project files to the newest formats.

What does this project look like?

❯ tree -P '*.sln|*.csproj'
.
└── src
    ├── SampleApp
    │   ├── Properties
    │   └── SampleApp.csproj
    ├── Turbocharged.Beanstalk
    │   ├── Properties
    │   ├── System
    │   └── Turbocharged.Beanstalk.csproj
    ├── Turbocharged.Beanstalk.Tests
    │   ├── FakesAndMocks
    │   ├── Properties
    │   └── Turbocharged.Beanstalk.Tests.csproj
    └── Turbocharged.Beanstalk.sln

9 directories, 4 files

Okay, so I’ve got one solution file with three csproj files.

That SampleApp is a WinForms project, so I’re probably not going to be able to convert it. I’ll ignore it for now.

❯ cd src

src❯ dotnet new sln --name Turbocharged.Beanstalk
Content generation time: 39.1996 ms
The template "Solution File" created successfully.

src❯  git status -s
 M Turbocharged.Beanstalk.sln

Git tells me that the solution file has been overwritten with the new format. Cool. But there are no projects in it yet. I need to convert the project files and add them to the solution.

The dotnet command can create several types of libraries, which I can see with dotnet new -all:

❯ dotnet new -all
Template Instantiation Commands for .NET Core CLI.

    ...

Templates                 Short Name       Language      Tags
-----------------------------------------------------------------------
Console Application       console          [C#], F#      Common/Console
Class library             classlib         [C#], F#      Common/Library
Unit Test Project         mstest           [C#], F#      Test/MSTest
xUnit Test Project        xunit            [C#], F#      Test/xUnit
ASP.NET Core Empty        web              [C#]          Web/Empty
ASP.NET Core Web App      mvc              [C#], F#      Web/MVC
ASP.NET Core Web API      webapi           [C#]          Web/WebAPI
Nuget Config              nugetconfig                    Config
Web Config                webconfig                      Config
Solution File             sln                            Solution

Examples:
    dotnet new mvc --auth None --framework netcoreapp1.1
    dotnet new nugetconfig
    dotnet new --help

So I need to create a class library and an xUnit project. I’ll also delete the skeleton source files that were generated:

src❯  dotnet new classlib --name Turbocharged.Beanstalk
Content generation time: 29.2166 ms
The template "Class library" created successfully.

src❯  dotnet new xunit --name Turbocharged.Beanstalk.Tests
Content generation time: 25.1647 ms
The template "xUnit Test Project" created successfully.

src❯ rm Turbocharged.Beanstalk.Tests/UnitTest1.cs Turbocharged.Beanstalk/Class1.cs

src❯  git status -s
 M Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj
 M Turbocharged.Beanstalk.sln
 M Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj

And now add those projects to my solution.

src❯ dotnet sln add Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj
Project `Turbocharged.Beanstalk.Tests/Turbocharged.Beanstalk.Tests.csproj` added to the solution.
Project `Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj` added to the solution.

Cool. Time to commit!

src❯ git commit -am 'convert csproj files to new project format'
[netstandard 61d7c86] convert csproj files to new project format
 3 files changed, 66 insertions(+), 225 deletions(-)

Making the class library buildable

Could it possibly…just work?

src❯ cd Turbocharged.Beanstalk
src/Turbocharged.Beanstalk❯ dotnet restore; dotnet build

   ...a lot of output...

obj/Debug/netstandard1.4/Turbocharged.Beanstalk.AssemblyInfo.cs(6,12): error CS0579: Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]
obj/Debug/netstandard1.4/Turbocharged.Beanstalk.AssemblyInfo.cs(7,12): error CS0579: Duplicate 'System.Reflection.AssemblyConfigurationAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]
obj/Debug/netstandard1.4/Turbocharged.Beanstalk.AssemblyInfo.cs(8,12): error CS0579: Duplicate 'System.Reflection.AssemblyDescriptionAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]
obj/Debug/netstandard1.4/Turbocharged.Beanstalk.AssemblyInfo.cs(9,12): error CS0579: Duplicate 'System.Reflection.AssemblyFileVersionAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]
obj/Debug/netstandard1.4/Turbocharged.Beanstalk.AssemblyInfo.cs(11,12): error CS0579: Duplicate 'System.Reflection.AssemblyProductAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]
obj/Debug/netstandard1.4/Turbocharged.Beanstalk.AssemblyInfo.cs(12,12): error CS0579: Duplicate 'System.Reflection.AssemblyTitleAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]
obj/Debug/netstandard1.4/Turbocharged.Beanstalk.AssemblyInfo.cs(13,12): error CS0579: Duplicate 'System.Reflection.AssemblyVersionAttribute' attribute [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]
NewtonsoftJsonJobSerializer.cs(6,7): error CS0246: The type or namespace name 'Newtonsoft' could not be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]
Trace.cs(12,25): error CS0246: The type or namespace name 'TraceSource' could not be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]
    0 Warning(s)
    9 Error(s)

Time Elapsed 00:00:01.74

No, of course not.

Most of these errors are from Properties/AssemblyInfo.cs. The new build system automatically generates properties that used to be specified in that file, so I don’t even need it anymore.

src/Turbocharged.Beanstalk❯ rm Properties/AssemblyInfo.cs

What’s next? Oh, I haven’t imported Newtonsoft.Json.

src/Turbocharged.Beanstalk❯ dotnet add package Newtonsoft.Json
Microsoft (R) Build Engine version 15.1.1012.6693
Copyright (C) Microsoft Corporation. All rights reserved.

  Writing /var/folders/kr/1g94089s3xv6vw34b9mgrq740000gn/T/tmpM2QiPo.tmp
info : Adding PackageReference for package 'Newtonsoft.Json' into project '/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj'.
log  : Restoring packages for /Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj...
info :   GET https://api.nuget.org/v3-flatcontainer/newtonsoft.json/index.json
info :   OK https://api.nuget.org/v3-flatcontainer/newtonsoft.json/index.json 410ms
info :   GET https://api.nuget.org/v3-flatcontainer/newtonsoft.json/10.0.2/newtonsoft.json.10.0.2.nupkg
info :   OK https://api.nuget.org/v3-flatcontainer/newtonsoft.json/10.0.2/newtonsoft.json.10.0.2.nupkg 385ms
info : Package 'Newtonsoft.Json' is compatible with all the specified frameworks in project '/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj'.
info : PackageReference for package 'Newtonsoft.Json' version '10.0.2' added to file '/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj'.

Finally, the last error is complaining about TraceSource. I’m sure there’s some replacement that works on .NET Standard, but I don’t know what it is. For now, I’ll just comment it all out.

Okay! Time to try again. I need to run dotnet restore first because I added a new package reference.

src/Turbocharged.Beanstalk❯ dotnet restore; dotnet build
  Restoring packages for /Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj...
  Writing lock file to disk. Path: /Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/obj/project.assets.json
  Restore completed in 266.09 ms for /Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj.

  NuGet Config files used:
      /Users/stephen/.nuget/NuGet/NuGet.Config

  Feeds used:
      https://api.nuget.org/v3/index.json
      /Users/stephen/.dotnet/NuGetFallbackFolder
Microsoft (R) Build Engine version 15.1.1012.6693
Copyright (C) Microsoft Corporation. All rights reserved.

PhysicalConnection.cs(56,25): error CS1061: 'TcpClient' does not contain a definition for 'Close' and no extension method 'Close' accepting a first argument of type 'TcpClient' could be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]

Build FAILED.

PhysicalConnection.cs(56,25): error CS1061: 'TcpClient' does not contain a definition for 'Close' and no extension method 'Close' accepting a first argument of type 'TcpClient' could be found (are you missing a using directive or an assembly reference?) [/Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/Turbocharged.Beanstalk.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:01.75

Hmm, TcpClient is missing the Close() method. They removed it in favor of Dispose(), so I just need to replace _client.Close() with _client.Dispose().

Does it build now?

src/Turbocharged.Beanstalk❯ dotnet build
Microsoft (R) Build Engine version 15.1.1012.6693
Copyright (C) Microsoft Corporation. All rights reserved.

  Turbocharged.Beanstalk -> /Users/stephen/src/github.com/jennings/Turbocharged.Beanstalk/src/Turbocharged.Beanstalk/bin/Debug/netstandard1.4/Turbocharged.Beanstalk.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.78

Yes! Commit before anything else goes wrong.

src/Turbocharged.Beanstalk❯ git commit -am "make Turbocharged.Beanstalk buildable"
[netstandard 1d5009c] make Turbocharged.Beanstalk buildable
 4 files changed, 15 insertions(+), 49 deletions(-)
 delete mode 100644 src/Turbocharged.Beanstalk/Properties/AssemblyInfo.cs

That’s enough for now.

Posted Saturday, June 17, 2017

Personal Names

I’m lucky because most software is designed for me. I never have a decision to make when I run across a form:

First name

Last name

But not every name follows the “FirstName LastName” convention. Some names are written family-name-first (China), some people have multiple family names (Hispanic cultures), and others have more complicated rules.

This article from the W3C is an excellent primer on how personal names differ across cultures.

The gist is that, if you’ve written this code:

Console.WriteLine($"Hello {firstName} {lastName}!");

then you might have called some people by the wrong name. If you can, it could be better to use “full name” and “nickname” fields:

Full name

What should we call you?

For a more humorous description of cultural differences in names: Falsehoods Programmers Believe About Names.

Posted Saturday, April 23, 2016