Control Videos Without Switching To The Browser (Draft Notes)

These are the draft notes for post [[/posts/2fwl2ezzkq9p][2fwl2ezzkq9p]]

They are not really in order and need to be edited.

Throwing them here for now in order to get the main post up without having to refine all this.

** Draft Overview

I've been learning Blender from [[https://www.youtube.com/c/ryankingart][Ryan King Art's Blender Tutorails]]. They're great.

There's one part of the experience that I don't like. Constantly having to switch back and forth between Blender and Chrome so I can pause and rewind parts to watch again. That has nothing to do with Ryan King Art. It's inherent in the YouTube tutorial experience.

That kind of friction is tough for me. It makes it really hard for me to get into any type of flow which makes it harder for me to learn and makes it more likely that I'll quit.

I set out to fix that.

My goal was to create keyboard shortcuts that play/pause, rewind, and fast-forward YouTube videos playing in Chrome.

Creating shortcuts in the [[https://support.apple.com/lt-lt/guide/shortcuts-mac/apdf22b0444c/mac][Mac Shortcuts App]] with those snippets above pulls that off.

It's a little thing. But it makes a big difference to me. Getting and staying in a flow while learning makes it way better on every level.


Trying Alfred and got this note the first time:

“Alfred 4.app” wants access to control “Google Chrome.app”. Allowing control will provide access to documents and data in “Google Chrome.app”, and to perform actions within that app.

"Required for Alfred to be able to send automation events to this application."


#+NOTES:

  • The way I trigger these is by holding down COMMAND-OPTION-SHIFT then clicking either J, K, and L to rewind, play/pause, or fast-forward, respectively. I don't actually hold all the keys though. My [[https://www.zsa.io/moonlander/][keyboard]] is programmable. So, I've got it setup so each action is just one button press under my left thumb.

  • The Play/Pause button is an element on the page and easy to target directly. I didn't find buttons for the forward and backwards skipping.

  • I dug through the code and figured out that it's possible to use this in the browser console to do the forward/backward skips: document.getElementsByClassName('html5-video-player')[0].seekBy(10) - Unfortunately, that doesn't work if you inject it from an OSA JXA JavaScript or AppleScript.

  • The reason the .seekBy(10) function doesn't work from the OSA script is that it's added to the <video> element by the YouTube javascript and [[https://bugs.chromium.org/p/chromium/issues/detail?id=543437#c4][Chrome runs AppleScript commands in an isolated world]]

  • I looked at using [[https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/fastSeek][HTMLMediaElement.fastSeek()]] but it's not supported on Chrome.

  • One way that does work is to send simulated keystrokes or [[https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent][KeyboardEvents]] for "j" and "l" to Chrome. The problem I have for that is that in order to make it work you have to switch to Chrome momentarily for it to work. While that's not a big deal it feels janky.

  • I finally found [[https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/currentTime][HTMLMediaElement.currentTime]] which can both get and set the playhead location. That's the solution in place here.

  • It's probably possible to use [[https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement][HTMLMediaElement]] instead of the button click approach. As long as what I've got keeps working I'm not going to worry about that.

  • It looks like there's not a direct way to tell if a video is playing in an HTMLMediaElement. [[https://youtu.be/fAW3VkaN2HY?t=57][This video]] has this snippet designed to make the determination: !!(thePlayer.currentTime > 0 && !thePlayer.paused && !thePlayer.ended && thePlayer.readyState > 2) that didn't work for me out of the box, but I may have flubbed typing it.

  • I originally setup a script that would send key presses for the fast forward and rewind (the play/pause was able to use a straight javascript button click). The simulated keypresses meant you had to jump to the app for a split second. I didn't like that so I dug into the code and figure out the javascript only approach (TKTKTK confirm it works and you don't need to activate chrome temporarily to make it work)

  • One of the rabbit holes I went down

  • The code targets the tab that's currently visible in the front most Chrome window. You'll need to make new version if you want to address videos in other tabs.

  • This could also be done in AppleScript. I'm trying to use only OSA JavaScript so making an AppleScript version is left as an exercise for the reader.

  • I found [[https://askubuntu.com/a/1228051][this solution]] for Ubuntu. It uses [[https://github.com/altdesktop/playerctl][playerctl]] which isn't available on the mac without compiling it yourself. (Which I'm not sure would work since it uses the [[https://specifications.freedesktop.org/mpris-spec/latest/][MPRIS D-Bus Interface]] which doesn't look is in a default Mac setup.

  • To setup Shortcuts I clicked on the Run JavaScript For Mac Automation action. It asked me for confirmation that I wanted to allow that the first time through.

  • Being able to inject JavaScript into web pages via a hotkey is super interesting to me. Figuring this out was a slog so I can't think of any ideas to use it at the moment, but I'm sure [[https://ideas.alanwsmith.com/][I'll come up with something]]

#+REFERENCES:

  • [[https://support.apple.com/lt-lt/guide/shortcuts-mac/apd163eb9f95/mac][Run a shortcut while working on your Mac]]

  • [[https://sites.google.com/a/chromium.org/dev/developers/applescript?visit_id=638009714849767207-4033809052&p=applescript&rd=1][The Chromium Projects - Information for Third-party Applications on Mac]]

  • [[https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/currentTime][HTMLMediaElement.currentTime - MDN Documentation]]

  • [[https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/fastSeek][HTMLMediaElement.fastSeek() - MDN Documentation]]

  • [[https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent][KeyboardEvent - MDN Documentation]]

  • [[https://bugs.chromium.org/p/chromium/issues/detail?id=543437#c4][Chromium Issue 543437: Javascript executed from Applescript doesn't have access to the page variables]]

  • [[https://www.youtube.com/watch?v=fAW3VkaN2HY][Video: JavaScript : How to tell if a video element is currently playing?]]

  • [[https://stackoverflow.com/a/10461709/102401][Seek to a point in html5 video]]

  • [[https://forum.keyboardmaestro.com/t/jxa-problem-name-of-application/5685][JXA problem — name of application]]

  • [[https://computers.tutsplus.com/tutorials/a-beginners-guide-to-javascript-application-scripting-jxa--cms-27171][A Beginners Guide to JXA, JavaScript Application Scripting]]

  • [[https://github.com/johnelm/node-jxa/issues/27][Issue with the JXA Application.currentApplication() Object #27]]

  • [[https://stackoverflow.com/questions/10213703/how-do-i-view-events-fired-on-an-element-in-chrome-devtools][How do I view events fired on an element in Chrome DevTools?]]

  • [[https://developer.mozilla.org/en-US/docs/Web/Events/Creating_and_triggering_events][Creating and triggering event - MDN Documentation]]

  • [[https://apple.stackexchange.com/questions/324411/applescript-to-run-javascript-on-chrome][Applescript to run javascript on chrome]]

  • [[https://peter.sh/experiments/chromium-command-line-switches/][List of Chromium Command Line Switches]]

  • [[https://github.com/JXA-Cookbook/JXA-Cookbook/issues/3][JXA-Cookbook - keystroke/using doesn't seem to work #3]]

  • [[https://www.chromium.org/developers/design-documents/][The Chromium Projects - Design Documents]]

  • [[https://www.chromium.org/developers/design-documents/applescript/][The Chromium Projects - AppleScript Support]]

  • [[https://apple.stackexchange.com/a/92666/7828][Applescript click automation in Google Chrome browser?]]

  • [[https://discussions.apple.com/thread/251621867][Sending Keystrokes to Inactive Applications with Applescript]]

  • [[https://stackoverflow.com/q/4996334/102401][Send a key code to an application without activating it first?]]

  • [[https://apple.stackexchange.com/a/84357/7828][Creating a keyboard shortcut to pause Youtube video in a Google Chrome tab]]

  • [[https://stackoverflow.com/a/67372789/102401][Is there a way to pause a YouTube video in Google Chrome using AppleScript?]]

  • [[https://apple.stackexchange.com/q/58234/7828][Override iTunes "media" keys (play, pause, etc) for Spotify?]]

  • [[https://askubuntu.com/a/1228051][Can I pause YouTube in Chrome from the command line?]]

  • [[https://github.com/altdesktop/playerctl][playerctl app]]

  • [[https://specifications.freedesktop.org/mpris-spec/latest/][MPRIS D-Bus Interface Specification]]

  • [[https://support.apple.com/guide/shortcuts/welcome/ios][Apple Shortcuts User Guide]]

  • [[https://support.apple.com/lt-lt/guide/shortcuts-mac/apdf22b0444c/mac][Apple - Intro to Shortcuts On Mac]]

  • [[https://www.zsa.io/moonlander/][Moonlander Keyboard]]

  • [[https://docs.rs/mpris/latest/mpris/][The Rust mpris crate]]

  • [[https://github.com/airtower-luna/mpris-python][MPRIS2 client in Python]]

  • [[https://developer.chrome.com/docs/apps/nativeMessaging/][Chrome Apps Native Messaging]]

  • [[https://forum.keyboardmaestro.com/t/send-media-keys-to-specific-application-trick/19480][Send media keys to specific application trick?]]

  • [[https://dougscripts.com/itunes/itinfo/keycodes.php][Doug's AppleScripts - System Events, Key Code and Keystroke]]

  • [[https://apple.stackexchange.com/a/36947/7828][How do I automate a key press in AppleScript?]]

  • [[https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/dispatchEvent][MDN Documentation: dispatchEvent]]

  • [[https://dev.to/mitchartemis/creating-a-global-configurable-shortcut-for-macos-apps-in-swift-25e9][Creating a global configurable shortcut for MacOS apps in Swift]]