Parse shortcodes in text widgets by default

[ionfish]

Currently, shortcodes are only parsed within post content. It would, to my mind, be a nice enhancement to allow them to be parsed from within text widgets as well. The implementation is trivial, so the only real question is what problems this might throw up.

[Denis-de-Bernardy]

experimenting this on my end with a plugin, and it's working fine. punting pending patch.

[sillybean]

Adds shortcode processing for text widgets

[sillybean]

Patch attached.

[jamescollins]

Patch against trunk r12722

[jamescollins]

I have attached a refreshed patch that causes shortcodes to be parsed in text widgets. It's similar to sillybean's patch, except it's for the latest (trunk) version.

It would be good to get this into 3.0 (unless anyone can think of any potential problems it may cause).

[jamescollins]

Moving to 3.0 milestone.

Given that post writers can use shortcodes in their posts/pages, I don't think there's any problems with allowing people to use them in text widgets.

[sillybean]

Freshened up the patch and added the shortcode_unautop filter.

[sillybean]

patched against r13006; adds shortcode_unautop

[technosailor]

-1

The text widget is semantically meant for TEXT. Yes, HTML could be included, but no I don't think it's proper to assume that a text widget should be parsing shortcodes too. That introduces the possibility of all kinds of nefariousness, particularly on WPMU/WPMS installs

[sillybean]

A lot depends on what you're using shortcodes for. In most of my installations, it serves as a macro -- it gets replaced with longer text passages that need to be stored and updated centrally (like a list of rules, where you wouldn't want multiple, conflicting copies floating around).

[jamescollins]

We added shortcode parsing in text widgets to our WPMU installs over 6 months ago, and it hasn't caused us any problems.

The whole idea of shortcodes is to provide a way to add fancy features (eg image galleries, shopping cart buttons, etc) without needing to give blog administrators the ability to insert arbitrary HTML code.

If a shortcode is already available in a page/post, I don't think there's any reason why the administrator shouldn't also be able to include the same shortcode in a sidebar (text) widget.

For plugins, it avoids the need to create a special sidebar widget just to replicate the functionality of a widget, which means a much less cluttered widgets screen.

I have seen many existing plugins that add this shortcode parsing ability to text widgets, so it would be good to make this behavior standard in WordPress to avoid any confusion.

[jamescollins]

Replying to jamescollins:

I have seen many existing plugins that add this shortcode parsing ability to text widgets, so it would be good to make this behavior standard in WordPress to avoid any confusion.

FYI, ​PollDaddy is a good example of a plugin where this was recently added.

[sillybean]

Two more use cases: ​BBCode and ​internal linking.

[chrisscott]

Replying to technosailor:

-1

The text widget is semantically meant for TEXT. Yes, HTML could be included, but no I don't think it's proper to assume that a text widget should be parsing shortcodes too. That introduces the possibility of all kinds of nefariousness, particularly on WPMU/WPMS installs

FWIW, wp.com already allows this.

[sillybean]

Only administrators have access to the widgets in the first place, so at some point I think we have to trust site owners not to break their own sandboxes.

[Denis-de-Bernardy]

Marking this as tested. I've been using similar code in one of my plugins for ages. it just works.

[hakre]

It does not work, just search google for something like First argument is expected to be a valid callback, 'shortcode_unautop' was given and you'll see that it won't work. Looks like the callback ignite routine is not clever enough to check if something is callable before actually calling it.

Secondly it looks like that there is need to ensure a certain thing is loaded in widgets before certain filter actually can be used.

Full error message for reference:
call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, 'shortcode_unautop' was given in /home/tcraig/public_html/yeastinfectioncenter/wp-includes/plugin.php on line 166

Some blogs already report this as usefull for current and then 3.0. I strongly doubt that information.

So, yes it's tested, but the tests are not successfull. In 3.0 it works in 2.9.2 it does not work it looks like.

[sillybean]

I just tested again in fresh copies of both 2.9.2 and 3.0 and it's working beautifully.

[hakre]

Then my report must be in error. But what I'm wondering is, when a text_widget relies on that the shortcode_unautop function to be available and it is not (as the error message states) what's happening then?

[sillybean]

Perhaps the user is trying to make it work in 2.8.x? It looks like the function was new in 2.9.0.

[jamescollins]

Replying to sillybean:

Perhaps the user is trying to make it work in 2.8.x? It looks like the function was new in 2.9.0.

shortcode_unautop was added in [12302] (2.9.0).

The patch seems to work fine. Not sure if it can make it into 3.0 due to future freeze though?

[dd32]

We're in feature freeze, Approaching Beta in the near future, This is an enhancement and is not a regression over the previous release. Moving to Future Release for now, with request on dev-feedback for other devs thoughts.

[nacin]

Punting due to feature freeze. Very temping 'worksforme' candidate because it can be done via a one-line plugin.

[nacin]

Correction, two-line plugin. Either way, the file headers take up more room.

[jamescollins]

I agree that it should be punted to 3.1 due to feature freeze.

Even though this functionality can be added with a simple 2 line plugin, I still think it should be added to core.

Shortcodes can be used in posts/pages, so I don't see why they can't be used in text widgets.

dd32, did you mean to add the dev-feedback tag to this ticket?

[dd32]

dd32, did you mean to add the dev-feedback tag to this ticket?

Indeed i did.. Just to track that this needs developer action.

[hew]

+1 for adding this to core.

[tmoorewp]

Refreshed patch against r18407.

+1 for adding to core. I already do this in a large MultiSite install and it is very useful.

[westi]

In [18592]:

Parse shortcodes in text widgets by default. Fixes #10457 props tmoorewp, sillybean, jamescollins.

[nacin]

Follow-up: #19411, do_shortcode() runs before wpautop() in the text widget.

[nacin]

Unfortunately there are too many holes that can be poked in this. For background, read #19411 and ​this IRC conversation.

Summarizing the problems as follows:

One, do_shortcode() would be running before our conditional wpautop() in the text widget. Shortcodes are used to running both after wpautop() and shortcode_unautop(). There are some ways around it, but it is going to be very hacky, and each hack sacrifices something else.

For example, currently anything running on the widget_text filter is always operating before wpautop(), so adding wpautop() to priority 10 will break some things, especially since plugins also on priority 10 would run after.

As a second example, I think do_shortcode() can only be run once against a block of text, otherwise it risks executing escaped shortcodes (not tested whether that is the case). So if a plugin is already adding do_shortcode() to widget_text, we could assume priority 10 and either only add with priority 10 (as trunk is now), or we'll need a delayed has_filter() check. This is why I left do_shortcode() on priority 10 and shoehorned wpautop() before it (for patches on #19411).

That's just the first problem. The second problem is what is resulting in a revert here. Core only has three shortcodes: embed, gallery, and caption. None of them are relevant for something other than a post. It looks like the caption shortcode would work (but is pretty much useless) and the gallery shortcode would just rely on whatever the post global is currently.

But, the embed code would fail. Hard. I haven't tested, but my impression would be that it would cache against the $post global. Since the specific post would constantly change for the contexts of a sidebar, the oembed response would get cached everywhere (first problem) and never be reliably cached, resulting in HTTP requests on every pageload (second problem).

Only plugin shortcodes can benefit from adding do_shortcode() to widget_text. Generally, we've used the guideline that until something benefits something that core does, it's probably not worth it. Worse, in this case, a core shortcode would result in bugs.

*

After writing all of this, upon further inspection of how the embed handlers register the [embed] shortcode, it can only work for posts, and will gracefully fail with __return_false for widgets.

However, I think there are enough problems here with wpautop/shortcodes to revert and deal with it in 3.4, given we are in RC and it will only benefit plugin shortcodes.

[nacin]

More IRC chatter:

AaronCampbell: For 3.3 I think the answer is revert
nacin: I agree. When we find gotchas like this in RC, we missed
	something previously, it's time to back it out.
AaronCampbell: Where I'm torn is for the future.  I think this adds a
	lot of flexibility to the text widget that didn't used to be there, but
	the fact that it doesn't benefit any core shortcodes makes it a tough call
nacin: I think adding it in the future is good, it's a nice-to-have and
	really helpful.
nacin: That said, shortcodes are used to looking for a $post global.
nacin: Which bothers me a bit.
nacin: It feels plugin material.

[nacin]

Patch thanks to an idea from AaronCampbell.

[nacin]

10457.3.diff​ avoids using the filter at all. Patch neglects to remove filters from default-filters but that's easy.

This feels hacky and lame, but would keep the feature alive for 3.3. We should have a new filter *after* widget_text we can attach things to, or something.

[westi]

Replying to nacin:

10457.3.diff​ avoids using the filter at all. Patch neglects to remove filters from default-filters but that's easy.

This feels hacky and lame, but would keep the feature alive for 3.3. We should have a new filter *after* widget_text we can attach things to, or something.

I would rather we revert and revisit the feature than do anything hacky to keep it alive.

I think we should probably take a step back and just think about how we can make the text_widget not so lame in general - This may mean things like using a CPT to store the data and giving you full WYSIWYG and shortcode functionality that way.

This was meant to be a quick win and it turns out it wasn't it's too late in the cycle to be hacking in fixes like this.

[westi]

In [19547]:

Revert [18592] because there are issues to solve with autop/shortcode parsing ordering before we add this. See #10457 and #19411

[nacin]

Replying to westi:

This was meant to be a quick win and it turns out it wasn't it's too late in the cycle to be hacking in fixes like this.

100% agree.

[nacin]

And punt.

[bobbingwide]

Hi, I'd just like to say that I've done quite a bit of work on shortcodes in my plugin called oik.

I wrote a set of functions that allow a shortcode to determine whether or not to expand depending on the value of the current filter. I've termed them smart shortcodes.

There were quite a few gotchas...
I had to write code to allow for the fact that do_shortcode could get invoked multiple times - catering for this by changing the '[' to &#91 when it was not wise to expand the code.

The wpautop()code has always been annoying, inserting unwanted breaks (<br />) in all sorts of places. My feeling is that it should NOT insert a break when the line ends with a shortcode ending.

To overcome the current behaviour I often have to put shortcodes side by side in the content. This is especially bothersome when the shortcodes expand into <div>s, <span>s and/or other generated XHTML.

I also had to add special code to support shortcode expansion in wp_footer, since current_filter() returns empty in this instance.

To cut a long story short, I agree that more work needs to be done.

[jane]

Yes, please. Totally fits into the 'make your site look the way you want it to' theme of 3.4.

[nacin]

Needs a new approach.

[SergeyBiryukov]

#27108 was marked as a duplicate.

[webdevmattcrom]

Chatted about this in Slack today. Here's how I see it happening (potentially), and would love for @nacin to elaborate on "Needs new approach"

• 1 -- Simply support shortcodes in Core Text Widgets and use the prevailing filter method (add_filter('widget_text', 'do_shortcode');`) to DISABLE them instead of enable them.
• 2 -- Add a checkbox in the Text Widget that simply says "Enable Shortcodes" directly under the existing "Automatically add Paragraphs" checkbox.
• 3 -- Add a setting somewhere under "Settings". Perhaps Reading...?

Of those three I think "2" makes the most sense because it fits with the current user experience and is not global.

Thoughts? Happy to contribute to this if it's a viable idea.

[nacin]

Replying to webdevmattcrom:

would love for @nacin to elaborate on "Needs new approach"

I'd read the whole ticket + the linked IRC conversation from 4 years ago.

[webdevmattcrom]

Thanks for your reply @nacin

I thought I had read everything (it's a lot of info here), and tried to reply carefully, but I see this from you in the IRC link now:

"Since no core shortcode is usable in widgets, I'm tempted to leave it to a plugin until we can solve the edge cases. It's useful, I agree, but it's a textbook situation for us leaving things to plugins that only benefit other plugins."

Also, considering all the effort going into shortcodes (​https://make.wordpress.org/core/2015/09/04/shortcode-roadmap-extended-discussion/) maybe it's best to revisit this after all that dust settles.

I will say that while Core shortcodes aren't supported in widgets, I don't believe that justifies not exploring this feature generally. Shortcodes are intended to be extensible, and therefore provide plugin authors the ability to create their own shortcodes. By not supporting shortcodes in Text Widgets by default, plugin authors have to create their own widgets (which is a good practice anyway) but that can lead to Widget overload for users who have a lot of plugins on their site.

Overall, having a clean and secure way of enabling shortcodes in widgets feels very developer-friendly and increases the reliability and security of WordPress rather than leaving that to third-party solutions.

[swissspidy]

#34361 was marked as a duplicate.

[westonruter]

Note that this came up again in introducing TinyMCE to the Text widget in 4.8. See dev note:

The same is true for shortcodes: they are not processed by default since many shortcodes presuppose a global $post as the context within which they expect to be running. Plugins may opt-in selectively to support individual shortcodes by filtering widget_text as they have had to do for many years now.

​https://make.wordpress.org/core/2017/05/23/addition-of-tinymce-to-the-text-widget/

[westonruter]

Patch developed for #40854.

[westonruter]

Milestoning this for 4.9 since shortcode support in the Text widget is required for supporting Add Media functionality: #40854

See 10457.4.diff​ which essentially captures the spirit of 10457.3.diff​, though it looks differently now with the Text widget now using TinyMCE.

Commit coming shortly. I believe I've accounted for all of the scenarios in unit tests. If there's anything I've missed, please re-open the ticket.

[westonruter]

In 41361:

Widgets: Add shortcode support inside Text widgets.

• Used now in core to facilitate displaying inserted media. See #40854.
• The [embed] shortcode is not supported because there is no post context for caching oEmbed responses. This depends on #34115.
• Add do_shortcode() to the widget_text_content filter in the same way it is added for the_content at priority 11, with shortcode_unautop() called at priority 10 after wpautop().
• For Text widget in legacy mode, manually apply do_shortcode() (and shortcode_unautop() if auto-paragraph checked) if the core-added widget_text_content filter remains, unless a plugin added do_shortcode() to widget_text to prevent applying shortcodes twice.
• Ensure that global $post is null while filters apply in the Text widget so shortcode handlers won't run with unexpected contexts.

Props westonruter, nacin, aaroncampbell.
See #40854, #34115.
Fixes #10457.

[westonruter]

Dev note: ​https://make.wordpress.org/core/2017/10/24/widget-improvements-in-wordpress-4-9/

[bobbingwide]

Re: ​https://make.wordpress.org/core/2017/10/24/widget-improvements-in-wordpress-4-9/
Here are some notes for those people who don't want to short-circuit when $post is null.

I have a number of shortcodes which adjust their behaviour depending on the values in $GLOBALS['post']. With 4.9-RC2, when these shortcodes are run in unchanged text widgets, they produce the wrong result.

I found that I needed to extend my code a little.

When looking for the global post id look in $GLOBALS['post'] first. If not set try $GLOBALS['id'].

When looking for the global post type, again look in $GLOBALS['post'].
If not available and $GLOBALS['id'] is set then call get_post( $GLOBALS['id'] ) and return the post_type from the reloaded post.

I understand that this problem does not occur with the Custom HTML widget.
I may use the Custom HTML widget in the future but since that requires more effort than simply updating a plugin it was not my first option.

[bobbingwide]

Replying to bobbingwide:

I understand that this problem does not occur with the Custom HTML widget.

Having thought about this a little longer I'd like to re-open this ticket.
As previously noted, I have shortcodes which make use of the context in which they're run.
With WordPress 4.9, those that are in text widgets will not have direct access to global $post
but those that are in custom HTML widgets will.

So if I had one of each widget on the same sidebar, one of the shortcodes would work as expected but the other wouldn't.

This is not backward compatible.

I would like to see code that nulled the global $post reverted.

Consider the alternative scenario, where the user tries a shortcode that wasn't previously expanded in widgets and the shortcode uses global $post or get_post().

There'll still be different results depending on which widget the shortcode's being run from.

It would be an odd situation where the expected results come from the text widget but the custom HTML widget produces the wrong results. Especially if the text widget's suggesting the user tried the custom HTML widget.

Notes:

• My actual example is this [bw_related post_type=shortcode_example ].
• My shortcode expansion logic is sensitive to whether or not the page is an archive.

P.S. I have already changed my plugin, but I imagine there are many others which have not changed.

[westonruter]

@bobbingwide so you're saying the Text widget and the Custom HTML widget are inconsistent here? Core does not apply shortcodes in the Custom HTML widget at all. The Custom HTML widget does apply the widget_text filter for compatibility with code that users may have copied from the legacy (non-visual) Text widget over to the Custom HTML widget, but in core itself no shortcode processing is done with this filter. Core applies shortcodes at widget_text_content not widget_text. Do you have a plugin that adds shortcode processing to the widget_text filter?

So the solution isn't to remove the nullification of the $post in the Text widget, but rather to add $post nullification to the Custom HTML widget. Shortcodes that use the $post as context but then cache the output for performance will behave very strangely if we don't unset $post. Not all shortcodes will work in a widget context. Instance, if you add a [gallery] shortcode without any arguments, in a post context it will grab all of the attachments of the current post. But in a widget there very well may not be a $post global, and this usage of the [gallery] won't work. So it's up to the plugin developer and the user to use the shortcodes in their appropriate context.

I suggest we open a ticket for 4.9.1 that adds nullification of the $post for the Custom HTML widget and we re-close this ticket.

[bobbingwide]

Replying to westonruter:

Core does not apply shortcodes in the Custom HTML widget at all.

Does that make sense? Should that not have been part of the implementation?

Do you have a plugin that adds shortcode processing to the widget_text filter?

I certainly do. And I didn't need to change it for 4.8 since the do_shortcode logic was enable just in time because of my existing filter. It's called oik.

Instance, if you add a [gallery] shortcode without any arguments, in a post context it will grab all of the attachments of the current post.

Yes, exactly. Had I done that in my shortcode enabled text widget then that's what gallery would have done.
So now 4.9 would break it.

So it's up to the plugin developer and the user to use the shortcodes in their appropriate context.

My shortcodes need to be able to obtain the post ID in order to determine the context. If you hide it I'll have to dig further into core to save the value I need for later.

I suggest we open a ticket for 4.9.1 that adds nullification of the $post for the Custom HTML widget and we re-close this ticket.

That could resolve the inconsistency but it means that 4.9 will be shipped with broken backward compatibility.

[westonruter]

Replying to bobbingwide:

Replying to westonruter:

Core does not apply shortcodes in the Custom HTML widget at all.

Does that make sense? Should that not have been part of the implementation?

It is intentionally not part of the core implementation. Shortcodes are for content. The presence of something that looks like a shortcode could mess up HTML if it is unintentionally converted.

Instance, if you add a [gallery] shortcode without any arguments, in a post context it will grab all of the attachments of the current post.

Yes, exactly. Had I done that in my shortcode enabled text widget then that's what gallery would have done.
So now 4.9 would break it.

In that case your shortcodes should be using get_queried_object() instead of get_post().

I suggest we open a ticket for 4.9.1 that adds nullification of the $post for the Custom HTML widget and we re-close this ticket.

That could resolve the inconsistency but it means that 4.9 will be shipped with broken backward compatibility.

Core never officially supported shortcodes in the Text widget until 4.9 in part because of the problems of the global $post and how different shortcodes rely on it in different ways (content, caching, context, etc). So if this is backwards-incompatible with some plugins, then I think that's just how it's going to have to be. Developers have the ​dev note to refer to on how shortcodes are now officially supported in core. If they need to post context for shortcodes, then they need to explicitly use get_queried_object() to opt-in for that. I'll update the dev note to mention that.

I've opened #42547 to add nullification of global $post to the Custom HTML widget.

[westonruter]

If the post is not nullified, consider these two cases case:

1) Someone adds a Text widget to the sidebar with a [gallery] shortcode in it, wanting to show the attachments from the current singular post. If this widget is shown on an index page, then it will get populated with whatever the last first post was in the The Loop, and have unpredictable results (first post because The Loop resets after it finishes).

2) Even on a singular template it will have an unreliable result. Someone also adds a widget that lists out posts from some category on the site. But they don't reset the $post after they do so, which is a very common issue. For example, this was just fixed in core even for the Recent Posts widget: #37312. (Correction: the Recent Posts widget isn't a problem here because it's regarding the global query and it does reset post data.) The result is that [gallery] would never show the attachments for the current queried post anyway if it relied on get_post(). It would only ever show the attachments from the last post in that widget that listed out posts from the category.

[westonruter]

I've reconsidered this second case (nullifying the $post even on a singular template). All we need to do is override it to get_queried_object() instead of null if is_singular(). See #42548.

[westonruter]

In 42185:

Widgets: Set global $post to current queried object instead of nullifying when is_singular() while applying filters (and shortcodes) in Text widget and (via plugin) Custom HTML widget.

Also prevent [gallery] shortcode from dumping out every attachment on the site when a containing Text widget is shown on an archive template.

Props westonruter, bobbingwide, joemcgill for testing.
See #10457.
Fixes #42548, #42547 for trunk.

[westonruter]

In 42186:

Widgets: Set global $post to current queried object instead of nullifying when is_singular() while applying filters (and shortcodes) in Text widget and (via plugin) Custom HTML widget.

Also prevent [gallery] shortcode from dumping out every attachment on the site when a containing Text widget is shown on an archive template.

Props westonruter, bobbingwide, joemcgill for testing.
See #10457.
Fixes #42548, #42547 for 4.9.