Parseval Proximal Neural Networks

Abstract

Abstract The aim of this paper is twofold. First, we show that a certain concatenation of a proximity operator with an affine operator is again a proximity operator on a suitable Hilbert space. Second, we use our findings to establish so-called proximal neural networks (PNNs) and stable tight frame proximal neural networks. Let $$\mathcal {H}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>H</mml:mi></mml:math> and $$\mathcal {K}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>K</mml:mi></mml:math> be real Hilbert spaces, $$b \in \mathcal {K}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>b</mml:mi><mml:mo>∈</mml:mo><mml:mi>K</mml:mi></mml:mrow></mml:math> and $$T \in \mathcal {B} (\mathcal {H},\mathcal {K})$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>T</mml:mi><mml:mo>∈</mml:mo><mml:mi>B</mml:mi><mml:mo>(</mml:mo><mml:mi>H</mml:mi><mml:mo>,</mml:mo><mml:mi>K</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:math> a linear operator with closed range and Moore–Penrose inverse $$T^\dagger $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msup><mml:mi>T</mml:mi><mml:mo>†</mml:mo></mml:msup></mml:math> . Based on the well-known characterization of proximity operators by Moreau, we prove that for any proximity operator $$\mathrm {Prox}:\mathcal {K}\rightarrow \mathcal {K}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>Prox</mml:mi><mml:mo>:</mml:mo><mml:mi>K</mml:mi><mml:mo>→</mml:mo><mml:mi>K</mml:mi></mml:mrow></mml:math> the operator $$T^\dagger \, \mathrm {Prox}( T \cdot + b)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msup><mml:mi>T</mml:mi><mml:mo>†</mml:mo></mml:msup><mml:mspace/><mml:mi>Prox</mml:mi><mml:mrow><mml:mo>(</mml:mo><mml:mi>T</mml:mi><mml:mo>·</mml:mo><mml:mo>+</mml:mo><mml:mi>b</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:math> is a proximity operator on $$\mathcal {H}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>H</mml:mi></mml:math> equipped with a suitable norm. In particular, it follows for the frequently applied soft shrinkage operator $$\mathrm {Prox}= S_{\lambda }:\ell _2 \rightarrow \ell _2$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>Prox</mml:mi><mml:mo>=</mml:mo><mml:msub><mml:mi>S</mml:mi><mml:mi>λ</mml:mi></mml:msub><mml:mo>:</mml:mo><mml:msub><mml:mi>ℓ</mml:mi><mml:mn>2</mml:mn></mml:msub><mml:mo>→</mml:mo><mml:msub><mml:mi>ℓ</mml:mi><mml:mn>2</mml:mn></mml:msub></mml:mrow></mml:math> and any frame analysis operator $$T:\mathcal {H}\rightarrow \ell _2$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>T</mml:mi><mml:mo>:</mml:mo><mml:mi>H</mml:mi><mml:mo>→</mml:mo><mml:msub><mml:mi>ℓ</mml:mi><mml:mn>2</mml:mn></mml:msub></mml:mrow></mml:math> that the frame shrinkage operator $$T^\dagger \, S_\lambda \, T$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msup><mml:mi>T</mml:mi><mml:mo>†</mml:mo></mml:msup><mml:mspace/><mml:msub><mml:mi>S</mml:mi><mml:mi>λ</mml:mi></mml:msub><mml:mspace/><mml:mi>T</mml:mi></mml:mrow></mml:math> is a proximity operator on a suitable Hilbert space. The concatenation of proximity operators on $$\mathbb R^d$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msup><mml:mi>R</mml:mi><mml:mi>d</mml:mi></mml:msup></mml:math> equipped with different norms establishes a PNN. If the network arises from tight frame analysis or synthesis operators, then it forms an averaged operator. In particular, it has Lipschitz constant 1 and belongs to the class of so-called Lipschitz networks, which were recently applied to defend against adversarial attacks. Moreover, due to its averaging property, PNNs can be used within so-called Plug-and-Play algorithms with convergence guarantee. In case of Parseval frames, we call the networks Parseval proximal neural networks (PPNNs). Then, the involved linear operators are in a Stiefel manifold and corresponding minimization methods can be applied for training of such networks. Finally, some proof-of-the concept examples demonstrate the performance of PPNNs.

Topics & Concepts