MRCC 880 issue with patch data sysex not transmitting from PC to synth DIN input

[WireWrangler]

SUBJECT: 
MRCC 880 issue with patch data sysex not transmitting from PC to synth DIN input

Using Conductive Labs MRCC 880
rcvd and tested 2022-10-29


ROUTING:
On the MRC 880, I have the routing set up correctly for
* DIN MIDI OUT to PC IN and
* PC OUT to DIN MIDI IN


=== Testing with a Casio CZ-101 synth ===
SUMMARY:
* DAW and CZ-101 work as expected when transmitting/receiving note/controller information, using MRCC 880

* PROBLEM: sysex from PC to CZ-101 not transmitting successfully in Patch Base patch editor application using MRCC 880, whereas this operation works perfectly with a simple USB to MIDI IN/OUT adapter


ENVIRONMENT:
* Hardware - Casio CZ-101, MRC 880, mac mini
* Software - macOS 11.7, Patch Base patch editor/librarian, Logic Pro, MIDI Monitor


DETAILS:
Have confirmed the following by using the macOS app "MIDI Monitor" and including in its Sources preferences "Spy on output to destinations" => so in effect, the app will display data going *into* and *out of* the PC

1. In the Patch Base app, I can use its virtual keyboard to successfully send Note On/Note Off to the CZ-101, and that data is reflected in MIDI monitor's data window. Just like a DAW, in that respect.

So, notes can be successfully routed to the CZ-101 DIN input from PC, using the MRCC 880.


2. I can use Patch Base's "Fetch" icon to successfully transfer the CZ's current patch to both MIDI Monitor app and Patch Base app -- 264 bytes of sysex -- and I can examine that data in detail in MIDI Monitor app. This is the sysex for 1 CZ patch => so sysex transfer from DIN output to PC input is working as expected.

To perform the dump from CZ to computer (initiated from computer), a dump request is first successfully sent from PC output to CZ DIN input (10 bytes): F0 44 00 00 70 10 60 70 31 F7.

=> So, at least *some* sysex is getting through to CZ DIN input from PC, just not patch data itself.


3. I can load a new patch in the Patch Base editor, from hard drive, and it will successfully populate the app's editor window (per usual) and its data will be reflected in MIDI monitor's data window as being sent to output destinations.

Doing so SHOULD automatically then update the synth's temporary patch memory to the new patch. It DOESN'T, when connected through MRCC 880; it DOES work when connected via a simple USB to MIDI IN/OUT adapter. This is consistent behavior, across many tests.

I can then make a change to the patch in Patch Base, and a complete batch of 264 sysex bytes will again be reflected in MIDI Monitor window, with each change -- i.e., the patch editor is dumping the entire patch's sysex to CZ, on each change. Again, such changes to a patch on the PC are NOT reflected in the synth's temporary patch memory, so no data is apparently getting through to the CZ.

During this operation, I can set the MRCC 880 to "MIDI Mon" and see a brief flash, so apparently something is coming into MRCC; but in any case, data is not then transferred successfully to the CZ-101, no matter which MIDI IN DIN port it is connected to.

- - -
Don't know if a firmware update can address this issue, but as of now, the new unit is not working as expected. I tried lowering the MIDI transmit speed to MRCC Ports in Patch Base -- no joy, same issue.

Thanks for any help. There is nothing on the forums I can see about this issue - nothing much yet about the MRCC 880, in general, as it has just been released.

[oldgearguy]

It may be that the 880 has a 256 byte limit to the size of a sysex message.  I know that was/still is an issue with the MRCC

[WireWrangler]

==> oldgearguy - Yes, that's it: MRCC 880 apparently has a 256 byte transmit limit. Thank you!

Prior to reading your reply I was playing around with transmitting the same CZ patches with "SysEx Librarian" app on macOS, and the app provides a preference for "transmit buffer size."

When changing the value to anything other than Default (i.e. 256, 128, 64. etc) the patch successfully transmitted the patch's 264 bytes and the synth's temporary memory was updated in real time. And after that, 7 bytes are now sent back to the PC from the synth (F0 44 00 00 70 30 F7).

Not sure what "Default" value is but it is obviously more than 256 (512?)

So, since the MRCC doesn't work for this application without being able to set this value in an application -- bummer -- it looks like my only other alternatives are to pursue with the app developer to provide a transmit buffer preference... or try to return the MRCC 880, since patch editing for legacy synths is its primary use, currently.

But at least I now know a way to transmit patches to the CZ's internal temporary memory. Thanks again.

[Darryl]

We have not seen this issue in testing.

We have tested with everything from 6 byte messages, all the way to 151KB firmware updates. And we've sent at least megabytes of SysEx data in testing to ensure reliability.

Most MIDI devices specify a SysEx buffer size because they can only receive certain size chunks of data at a time. Sometimes they need that data to be throttled as well. That is something configured in the software used to send the data. If you set that value to something the synth can't consume, it won't work. The MRCC 880 has internal buffers, but this does not have a relation to the above mentioned limitations of the receiving device. 

Here are some screens from MIDI OX I just made, sending and receiving a 37K file in 1024 byte chunks. Sent via MRCC 880 USB 1, routed to DIN 1 Out, connected to an MRCC XpandR which MIDI OX receives as the input for a round trip.
           

[WireWrangler]

We have not seen this issue in testing.

We have tested with everything from 6 byte messages, all the way to 151KB firmware updates. And we've sent at least megabytes of SysEx data in testing to ensure reliability.

Most MIDI devices specify a SysEx buffer size because they can only receive certain size chunks of data at a time. Sometimes they need that data to be throttled as well. That is something configured in the software used to send the data. If you set that value to something the synth can't consume, it won't work. The MRCC 880 has internal buffers, but this does not have a relation to the above mentioned limitations of the receiving device. 

Here are some screens from MIDI OX I just made, sending and receiving a 37K file in 1024 byte chunks. Sent via MRCC 880 USB 1, routed to DIN 1 Out, connected to an MRCC XpandR which MIDI OX receives as the input for a round trip.

Well, all I can say is that I have tested this extensively -- albeit with this one CZ synth that sends/receives 260 bytes per patch -- and I can say for certain that on macOS the "transmit buffer size" setting in SysEx Librarian -- set at 256 bytes or less -- was what enabled a patch to successfully be transferred from PC to synth with the MRC 880.

And before changing that preference in SysEx Librarian -- it is set to Default typically, which is apparently greater than 256 bytes -- its behavior was exactly like that of the other macOS app I used to test: Patch Base.

And I can also say that all other MIDI interfaces I use on macOS transmit CZ  patches back and forth and work as expected: it's only the 880 the doesn't, and it only has issues from PC to DIN.

Having said all of that, I need to test other devices with sysex and the MRC 880.

And as I said in another thread, otherwise, I really like the MRC 880 - it is sort of what we have always needed for "desktop lab" connections, which is how I am currently deploying it. Thanks for producing it and offering for sale at an affordable price.

---

We have not seen this issue in testing.

We have tested with everything from 6 byte messages, all the way to 151KB firmware updates. And we've sent at least megabytes of SysEx data in testing to ensure reliability.

Most MIDI devices specify a SysEx buffer size because they can only receive certain size chunks of data at a time. Sometimes they need that data to be throttled as well. That is something configured in the software used to send the data. If you set that value to something the synth can't consume, it won't work. The MRCC 880 has internal buffers, but this does not have a relation to the above mentioned limitations of the receiving device. 

Here are some screens from MIDI OX I just made, sending and receiving a 37K file in 1024 byte chunks. Sent via MRCC 880 USB 1, routed to DIN 1 Out, connected to an MRCC XpandR which MIDI OX receives as the input for a round trip.

Well, all I can say is that I have tested this extensively -- albeit with this one CZ synth that sends/receives 260 bytes per patch -- and I can say for certain that on macOS the "transmit buffer size" setting in SysEx Librarian -- set at 256 bytes or less -- was what enabled a patch to successfully be transferred from PC to synth with the MRC 880.

And before changing that preference in SysEx Librarian -- it is set to Default typically, which is apparently greater than 256 bytes -- its behavior was exactly like that of the other macOS app I used to test: Patch Base.

And I can also say that all other MIDI interfaces I use on macOS transmit CZ  patches back and forth and work as expected: it's only the 880 the doesn't, and it only has issues from PC to DIN.

Having said all of that, I need to test other devices with sysex and the MRC 880.

And as I said in another thread, otherwise, I really like the MRC 880 - it is sort of what we have always needed for "desktop lab" connections, which is how I am currently deploying it. Thanks for producing it and offering for sale at an affordable price.

The other obvious thing is that the CZ-101 is an ancient device and very likely "can only receive certain size chunks of data at a time" -- as you say.

What mystifies me, though, is why the two pieces of macOS sysex software mentioned previously work perfectly with this ancient synthesizer and any other MIDI router/interface I have used other than the MRCC 880?  

My uninformed intuition says 'some sort of timing issue' -- e.g. it might be necessary to build in a delay between chunks of data transmission, as sometimes occurs in software -- but I have no idea in this context, and I would really appreciate a more informed point of view about any practical way to deal with this on the MRCC 880. 

I have already contacted the software author to see if it is possible to build in a pref for SysEx buffer size.

[Darryl]

I downloaded a CZ-101 patch (attached) and sent it the same way as above. Its working as expected, so I don't really know how to troubleshoot the issue. I tried with MIDI OX set both for a 256 byte output buffer, and 266 byte output buffer.

       

On the MIDI OX input, we can see when set for 256 byte output buffer, the patch sends in 2 parts. With a 266 byte buffer, which is larger than the actual patch size, it sends all in one message of the correct size. Either way, it shouldn't look any different with another MIDI interface, and I have no idea why the synth wouldn't receive it.

By the way, the MRCC 880 doesn't blink the LEDs for SysEx data. I'm thinking about what else we can try... but any bright ideas are welcome :-)

I've just read this page, its complicated to decipher but it indicates that there is two way communication between the CZ and the software. 
http://www.youngmonkey.ca/nose/audio_tec...io-CZ.html

Does the CZ need the output of the synth connected to the input of the computer to complete messages?

[WireWrangler]

I downloaded a CZ-101 patch (attached) and sent it the same way as above. Its working as expected, so I don't really know how to troubleshoot the issue. I tried with MIDI OX set both for a 256 byte output buffer, and 266 byte output buffer.



On the MIDI OX input, we can see when set for 256 byte output buffer, the patch sends in 2 parts. With a 266 byte buffer, which is larger than the actual patch size, it sends all in one message of the correct size. Either way, it shouldn't look any different with another MIDI interface, and I have no idea why the synth wouldn't receive it.

By the way, the MRCC 880 doesn't blink the LEDs for SysEx data. I'm thinking about what else we can try... but any bright ideas are welcome :-)

I've just read this page, its complicated to decipher but it indicates that there is two way communication between the CZ and the software. 
http://www.youngmonkey.ca/nose/audio_tec...io-CZ.html

Does the CZ need the output of the synth connected to the input of the computer to complete messages?

Thanks for doing all of that. I really appreciate it.

Don't know how to account for the difference in behavior among MIDI interfaces/routers -- and MIDI OX on Windows and various MIDI apps on macOS -- other than the fact that they are 'different' hardware and software with different OSs and APIs.  This is not a level of programming that I am familiar with - I mean I used to write MIDI apps with high level scripting tools; and low level machine language on a very ancient 6502 processor, but never had to account for SysEx issues with arcane and archaic synthesizers.

Yes, on the MRCC 880, lights don't blink with incoming SysEx, but they do light up... so that's something, at least, to see the routing.

Yes, there is two way communication with the CZ and computer in data dumps.

Yes, I have seen reference to youngmonkey previously as a credit on Patch Base developer's website. Yes, this is some convoluted *&%^ re: CZ's sysex. Somehow, Patch Base's developer deciphered all of that to make a great patch editor - kudos.

And yes, I think the CZ needs the output of synth to 'complete the messages' in some respect, anyway.

What I do know: there appears to be two way communication, from both directions. I saw it when there was a successful transfer from PC to CZ, on the Mac:

1. PC sends 264 bytes to CZ
2. CZ sends 7 bytes back - acknowledgement? => not sure this part is absolutely necessary, but I definitely saw it in MIDI Monitor app, on *successful* transfer. On unsuccessful transfer, only step 1 (PC to CZ) - no ack.

And as I remember, when initiating a dump from synth to PC:

1. PC sends 10 bytes (dump request)
2. Synth dumps back to PC 

=> this is standard behavior for PC initiating dump request, yes?
And again, for the record: with MRCC 880 and Casio CZ-101, this direction ALWAYS works.

- - -
 As to bright ideas, all I could think of, beyond requesting the ability to set lower buffer size in software preferences (as currently exists in macOS SysEx Librarian), was to add a re-transmission filter between PC and MRCC 880; or in an intermediary computer app, a type of MIDI pipe with filtering -- to re-buffer data and transmit with lower transmit buffer size, suitable for the synth's needs. 

I do not know if this is even possible, nor have I not kept up with any commercial products that might address the need; but it seems kind of silly to do something like that here, when I have already confirmed that setting lower buffer size in software addresses the issue, when used with the MRCC 880. 

It would nice, on reflection, if "Audio MIDI Setup" on macOS allowed setting transmission speed and transmission buffer size universally, per port. Not going to happen, likely.

[oldgearguy]

I have an 880. I'll try to get some sysex testing in tonight or tomorrow evening.

Just so I can be clear -- what is the official Conductive Labs status for system exclusive transmission on both the 880 and the MRCC?

DIN <--> DIN, (both)
DIN <--> PC port. (both)
DIN <--> USB (MRCC)
USB <--> PC port (MRCC)

[WireWrangler]

Darryl wrote:
"By the way, the MRCC 880 doesn't blink the LEDs for SysEx data."

Yes, sorry for stating otherwise. Based on 880's observed behavior today, I can see that you are correct. No lights with SysEx, period.

=== CZ-101 has special needs re: transmit buffer size ===
From reading other posts online, I can confirm that CZ-101 needs extra help in SysEx transmission,
e.g., from a comment re: another MIDI software product, on GitHub:

'[...]  should have the ability to modify SysEx buffer size and the delay between the buffers being sent.

This is important for "legacy" devices such as the Casio CZ-101 which requires smaller packets of data and increased delay time for SysEx patch interchange to complete successfully.

This may also prove to be useful on other, older, MIDI devices which should not be left behind, even if they don't conform 100% to the MIDI specification.[...]'

So there you go.

=== Summary ===
Using macOS SysEx Librarian, with MRCC 880: 
* PC-CZ transfer doesn't work when "Transmit Buffer Size" is set to "Default" speed
* PC-CZ transfer DOES work when "Transmit Buffer Size"  is set to 256 bytes or less

Whereas, using macOS SysEx Librarian with other MIDI interfaces: 
* With an E-MU XMidi1X1 (simple USB to MIDI interface) =>  PC-CZ transfer works fine at "Default" speed
* With an iConnectivity iConnectAudio4 MIDI interface =>  PC-CZ transfer works fine at "Default" speed

=== Addendum (further testing with E-MU interface and MRCC 880) ===
Just for fun, I connected the E-MU MIDI interface with a single DIN IN/OUT, directly to PC via the E-MU's USB port, and then I connected the E-MU's DIN plugs to the 880 on DIN port 1 (in/out).

Then I connected the CZ-101 to DIN port 2 (in/out).

Created routing on the MRCC-880:
* MIDI IN 1 => MIDI OUT 2
* MIDI IN 2 => MIDI OUT 1

So in this scenario, the 880 is just using DIN ins and outs. 

Works perfectly with all macOS apps.  So this is a viable method of using the 880 with PC and apps accessing CZ-101 sysex.  Just not with the 880's USB port connected to PC - for some types of SysEx data transmission.

So again: the issue appears to be the 880's USB port transmission speed (and/or delay time, etc) from PC=>CZ and the CZ's inability to handle such, without adjusting "Transmit buffer speed" on an individual PC app.

Testing with at least four more legacy synths remains to be done.

---

I have an 880.  I'll try to get some sysex testing in tonight or tomorrow evening.

Just so I can be clear -- what is the official Conductive Labs status for system exclusive transmission on both the 880 and the MRCC?

DIN <--> DIN, (both)
DIN <--> PC port. (both)
DIN <--> USB (MRCC)
USB <--> PC port (MRCC)

Thank you for any testing you are able to do.  I will need to test additional synths/modules with the 880...

[Darryl]

Thanks for sharing that. Its completely normal to have to configure the buffer size of the sending software to the size the MIDI device requires. Any patch management software like Patch Base must comprehend the correct size chunks for each device they support, and for each type of data they can transfer.

More boring details for those that are interested...
Having a "global" sysex size wouldn't work for most devices because most devices, especially of the vintage variety, have a specific maximum size chunk of sysex they can consume. That size can be different depending on what kind of data is being sent. As an example, a DSI Poly Evolver wants the software sending SysEx to send 288 byte messages for patches, 1024 byte chunks for their "combo patches" file, and 128 byte chunks for a "Main" firmware update. Unlike the CZ, the Evolver has no flow control or ack.

The DSI Evolver "All Programs" sysex file that contains all four banks of stock patches has a leading F0 (start of sysex message) and trailing F7 (end of sysex message) for each patch in the file, so you could say its pre-chunked up into 288 byte messages. While the Evolver's 151KB "Main" firmware file has an F0 at the beginning and F7 at the end. They don't chunk it up in the file, so the sending software has to send it in 128 byte chunks (according to DSI's instructions). Sending the firmware in larger chunks would probably over-run the Evolver's input buffer, or exceed the speed at which they can write the data to flash storage.

All this to say, how SysEx is formatted is left to the device maker, and as a MIDI router, all we know is there's an F0 at the beginning, a bunch of data then an F7 when its done. The reason that a MIDI router has to care about this at all is because its doing much more processing than a 1 input, 1 output USB MIDI interface that just shoves the bytes through; it must examine every incoming byte of MIDI data; for filtering, to mix SysEx with MIDI real time data or merged data (and not other data), and it must have some "policy" about what gets prioritized when large chunks of SysEx data is sent to a merged output.

That policy for large SysEx is documented in the MRCC 880 User Guide.. here's a excerpt.
"Care must be taken when sending SysEx messages larger than 128 bytes while merging MRCC 880 inputs. When merging inputs, small SysEx messages, up to 128 bytes long will merge nicely with Realtime and other MIDI data, minimizing delay of Note On and Off messages. However, MRCC 880 will give priority to SysEx data exceeding 128 bytes, potentially blocking MIDI data on the other merged inputs."

In practice, when we tested sending moderately larger than 128 byte chunks of SysEx while merging, we did not "hear" delays or block data. It works as seamlessly as a 1 in, 1 out MIDI interface. But if you send a 100K+ SysEx firmware update, it will block other MIDI messages at some point when all the buffers from merged inputs are full. I don't know why anyone would be jamming while sending a firmware update or patch dump, but our advice is, don't do that. For a critical SysEx transfer like a firmware update or full patch dump, turn off other MIDI controllers so they aren't sending clock or active sense messages, etc. Or temporarily un-route inputs other than the one input receiving SysEx. Though it would probably be ok to let the MIDI router block this excess data, its probably a good idea to be cautious in this scenario.